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Abstract 


This  paper  documents  a  research  project  to  examine  the  relationship  between 
major  defense  program  managers’  attributes  and  the  outcomes  of  their  programs.  It 
develops  a  model  based  on  the  research  hypothesis  that  program  manager  (PM) 
professionalism  is  positively  correlated  with  program  outcomes.  Dependent 
variables  consist  of  metrics,  such  as  cost  variance,  that  indicate  program  outcomes. 
Independent  variables  consist  of  PM  attributes,  such  as  tenure  duration,  along  with 
several  extraneous  variables,  such  as  the  program  commodity  (e.g.,  ships,  aircraft). 
Data  were  obtained  from  Selected  Acquisition  Reports  for  major  defense  acquisition 
programs  for  the  years  1997-2010.  Expert  interviews  were  recorded  and  used  to 
frame  various  aspects  of  the  analysis  and  conclusions.  No  significant  correlation 
was  found  between  any  of  the  independent  variables  and  program  outcomes.  The 
findings  suggest  that  (1 )  further  research  is  needed  to  determine  good  predictors  of 
program  outcomes;  (2)  policy-makers  should  look  to  sources  other  than  the  PM  as  a 
cause  of  poor  outcomes;  and  (3)  a  contingency  approach  to  PM  career  management 
is  appropriate. 

Keywords:  Defense  acquisition;  program  managers;  program  management; 
professionalism;  program  outcomes 
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Executive  Summary 


The  professionalism  of  defense  acquisition  program  managers  (PMs)  has 
been  an  important  issue  in  policy  discussions  on  defense  reform  for  the  past  fifty 
years.  Numerous  sources  have  identified  the  lack  of  professional  qualifications 
(e.g.,  training,  education,  experience)  among  PMs  as  a  contributing  factor  to 
unfavorable  outcomes  such  as  cost  overruns  and  schedule  slips.  In  1990,  Congress 
enacted  the  Defense  Acquisition  Workforce  Improvement  Act  (DAWIA)  with  the  goal 
of  professionalizing  the  workforce,  including  PMs,  to  help  achieve  acquisition  reform. 

Such  actions  presume  that  enhanced  PM  attributes  contribute  to  improved 
program  outcomes,  yet  no  studies  have  attempted  to  validate  this  presumption.  This 
paper  documents  a  research  project  to  examine  the  relationship  between  PM 
attributes  and  the  outcomes  of  their  programs.  The  project  develops  a  model  based 
on  the  research  hypothesis  that  PM  professionalism  is  positively  correlated  with 
program  outcomes,  and  it  conducts  appropriate  statistical  testing  and  analysis. 
Dependent  variables  consist  of  metrics,  such  as  cost  variance,  that  indicate  program 
outcomes.  Independent  variables  consist  of  PM  attributes,  such  as  tenure  duration, 
along  with  several  extraneous  variables,  such  as  the  program  commodity  (e.g., 
ships,  aircraft).  Data  were  obtained  from  Selected  Acquisition  Reports  for  major 
defense  acquisition  programs  for  the  years  1997-2010.  Expert  interviews  were 
recorded  and  used  to  frame  various  aspects  of  the  analysis  and  conclusions. 

The  findings  provide  insights  into  policies  to  pursue  increased  PM 
professionalism  as  a  component  of  acquisition  reform.  Specifically,  no  significant 
correlation  was  found  between  any  of  the  independent  variables  and  program 
outcomes.  The  findings  suggest  that  (1 )  further  research  is  needed  to  determine 
good  predictors  of  program  outcomes;  (2)  policy-makers  should  look  to  sources 
other  than  the  PM  as  a  cause  of  poor  outcomes;  and  (3)  a  contingency  approach  to 
PM  career  management  is  appropriate. 


-  XI  - 
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Introduction 


For  over  fifty  years,  the  professionalism1  of  program  managers  (PMs)  charged 
with  the  acquisition  of  major  weapon  systems  has  been  an  important  issue  in  policy 
discussions  on  defense  reform  in  the  U.S.  (Baumgartner,  1979;  Denny,  1985;  Fox, 
1974,  1984,  1988;  Ladner,  1983;  Peck  &  Scherer,  1962). 2  Numerous  studies  and 
reports  from  academia,  the  Department  of  Defense  (DoD),  General  Accounting  Office 
(now  Government  Accountability  Office  [GAO]),  Congressional  Research  Service 
(CRS),  and  various  commissions  have  identified  the  lack  of  professional  qualifications 
(e.g.,  training,  education,  experience)  among  PMs  as  a  contributing  factor  to 
undesirable  acquisition  program  outcomes  such  as  cost  overruns  and  schedule  slips 
(Blue  Ribbon  Commission,  1986;  DoD,  1989;  Lockwood,  1985). 

To  address  this  issue,  Congress  during  the  1980s  and  1990s  enacted 
legislation — most  notably,  the  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA;  1990) — designed  to  enhance  professional  attributes  of  DoD  PMs 
(Lockwood,  1986,  1990;  Mavroules,  1991;  Snider,  1996).  These  actions  established 
standards  for  a  variety  of  PM  attributes  and  qualifications  such  as  minimum  levels  of 
training,  education,  experience,  and  tenure-in-office  requirements  to  reduce  PM 
turnover.  Concern  with  this  issue  has  also  been  evident  in  studies  that  seek  to 
determine  desired  PM  attributes  and  competencies  to  be  imparted  through  training, 
education,  or  experience,  with  the  objective  of  improving  acquisition  outcomes 
(Crawford,  2002;  Cullen  &  Gadeken,  1990;  Gadeken,  1994,  2002;  McVeigh,  1994; 
Tragar,  Hausmann,  &  Sayala,  2008). 


1  “Professionalism”  and  “professional”  in  this  report  refer  to  standards,  characteristics,  and  attributes 
of  persons  or  groups  with  specialized  knowledge,  skills,  and  experience  required  for  membership  in  a 
profession.  “Professionalize”  and  “professionalization”  refer  to  means  by  which  one  is  made 
professional.  While  few  argue  that  acquisition  or  program  management  is  a  profession,  most  agree 
that  professionalism  is  desirable  for  managers  of  defense  acquisition,  as  this  report  will  show. 

2  This  report  presumes  that  the  reader  has  working  knowledge  of  the  main  structural  features  of  the 
U.S.  defense  acquisition  system,  including  program  management,  and  is  familiar  with  the  long¬ 
standing  criticisms  of  its  performance. 
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A.  Issue 


Despite  these  measures  to  improve  PM  professionalism,  observers  continue 
to  judge  that  acquisition  outcomes  generally  have  not  improved  (see,  for  example, 
GAO,  2010a,  2011a).  At  the  same  time,  studies  continue  to  point  out  the  need  for 
enhanced  PM  professionalism  as  a  component  of  acquisition  reform  (Assessment 
Panel,  2005;  Defense  Business  Board  [DBB],  201 1).  This  apparent  failure  of  these 
efforts  calls  into  question  the  conventional  wisdom  regarding  the  extent  to  which  PM 
professionalism  influences  the  outcomes  of  very  large,  complex,  and,  because  of 
their  large  budgets,  often  politically-charged  defense  acquisition  programs.  Indeed, 
this  researcher  has  found  no  studies  that  support  the  presumption  that  PM 
professionalism  is  related  to  program  outcomes. 

B.  Purpose 

This  report  documents  a  research  project  to  examine  the  conventional 
wisdom  surrounding  PM  professionalization,  specifically,  the  view  that  PM  attributes 
are  related  to  program  outcomes.  Essentially,  the  project  entails  development  of  a 
model  based  on  the  research  hypothesis  that  PM  professionalism  is  positively 
correlated  with  program  outcomes,  along  with  appropriate  statistical  testing  and 
analysis.  Dependent  variables  consist  of  several  metrics,  such  as  cost  variance, 
that  indicate  program  outcomes.  Independent  variables  consist  of  PM  attributes, 
such  as  duration  in  office,  along  with  several  extraneous  variables,  such  as  the 
program  commodity  (e.g.,  ships,  aircraft,  missiles).  Data  were  obtained  from  a 
database  of  annual  Selected  Acquisition  Reports  (SAR)  for  major  defense 
acquisition  programs  for  the  years  1997-2010.  Expert  interviews  were  recorded  and 
used  to  frame  various  aspects  of  the  analysis  and  conclusions. 

C.  Benefit 

To  the  extent  that  the  research  hypothesis  (conventional  wisdom)  may  be 
supported,  the  extent  to  which  the  DoD  has  taken  a  proper  approach  in  pursuing 
increased  PM  professionalism  as  a  component  of  acquisition  reform  may  also  be 
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shown.  Findings  of  positive  correlations  between  PM  professionalism  and  program 
outcomes  may  indicate  that  even  further  investments  to  enhance  PM  qualifications 
may  be  warranted.  On  the  other  hand,  if  PM  qualifications  are  uncorrelated  with 
program  outcomes,  it  may  be  concluded  that  meaningful  reform  is  possible  only 
through  other  means,  for  example,  radical  overhaul  of  the  entire  acquisition  system 
(see,  for  example,  Lockwood,  1985,  p.  5). 

D.  Organization 

Following  this  introduction,  the  report  provides  a  policy  and  literature  review 
that  documents  in  more  detail  the  historical  context  of,  legislation  related  to,  and 
relevant  scholarship  on  the  issue  of  PM  professionalization.  The  report  then  turns  to 
develop  various  hypotheses  and  models  that  are  suggested  by  that  discussion,  and 
it  describes  the  data  to  be  employed.  It  then  presents  the  statistical  analyses  and 
provides  interpretation  of  the  results.  The  report  concludes  with  a  summary  of 
findings  and  policy  recommendations. 

E.  Challenges 

This  project  entails  several  challenges  that  should  be  acknowledged  at  the 
outset.  First,  defense  acquisition  represents  a  controversial  public  policy  issue 
around  which  there  is  little  consensus  (except,  perhaps,  on  the  need  for  reform). 
Thus,  any  models  and  data  that  purport  to  capture  and  describe  factors  contributing 
to  program  outcomes  are  subject  to  criticism.  Second,  the  fidelity  of  SAR  data  may 
be  questioned,  since  it  is  reported  by  program  management  offices  and  not 
automatically  collected  nor  audited.  Third,  the  models  in  this  project  should 
reasonably  and  logically  attempt  to  account  for  the  contributions  of  PM  attributes  to 
program  outcomes.  One  would  expect,  however,  that  those  contributions  would  not 
necessarily  be  manifested  in  any  single  time  period,  but  rather  may  occur  both  while 
the  PM  is  in  office  and  after  the  PM  has  left  office.  Some  simplifying  assumptions 
are  necessary  to  take  this  into  account. 
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Policy  Review 


This  section  provides  a  background  of  policy  pertaining  to  the  issue  of  PM 
professionalism.  The  policy  section  consists  of  summaries  of  selected  relevant 
legislation  and  reports  by  commissions,  committees,  and  bodies  like  the  GAO  and 
CSR;  these  are  presented  roughly  in  chronological  order. 

A.  The  Second  Hoover  Commission 

From  1953  to  1955,  the  second  Hoover  Commission’s  19  task  forces  and 
committees  conducted  a  comprehensive  review  of  federal  executive  branch 
functions,  organization,  and  policies.  Their  evaluation  criteria  emphasized 
preservation  and  enhancement  of  Constitutional  concerns  such  as  rule  of  law, 
separation  of  powers,  Congressional  power  of  the  purse,  and  civilian  control.  They 
also  addressed,  however,  the  economic  and  efficient  operation  of  federal  bureaus, 
reflecting  taxpayer  concerns  with  waste  and  duplication  of  effort  which  the  first 
Hoover  Commission  had  documented  (MacNeil  &  Metz,  1956,  pp.  13-17). 

When  asked  which  of  the  Commission’s  314  recommendations  he  would 
judge  as  the  single  most  critical,  Hoover  named  the  recommendation  to  set  up  a 
senior  civil  service  (MacNeil  &  Metz,  1956,  p.  29).  His  choice  reflected  the  view  that, 
considering  the  growing  complexities  of  20th  century  governance,  professional 
administrators  were  essential  to  achieve  the  reconciliation  of  efficient  management 
within  a  Constitutional  framework  of  law  and  politics.  This  perspective  is  evident  not 
only  in  the  Commission’s  recommendations  on  managers  for  defense  acquisition,3 
but  also  in  almost  all  legislative  and  commission  recommendations  regarding  PMs 
since  then. 


3  Because  acquisition  had  not  yet  entered  general  usage,  the  Commission  used  terms  like  supply  and 
research  and  development  to  refer  to  acquisition-related  activities. 
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The  need  for  program  management.  The  Commission  noted  that  the  DoD’s 
organizations  and  processes  were  ill-suited  for  the  acquisition  of  20th  century 
weaponry  and  that,  due  to  complexities  of  requirements  and  operations,  a  “single 
integrated  program”  (MacNeil  &  Metz,  1956,  p.  245)  is  required  to  develop  and 
produce  a  technically  advanced  defense  system.  This  view  reflects  the  emergence 
of  project  management  (thus  also  the  position  and  role  of  the  PM)  during  these 
years  as  a  discipline  for  managing  complex  and  unique  acquisition  efforts,  such  as 
the  Manhattan  Project  and  development  of  the  first  U.S.  intercontinental  ballistic 
missiles  (Baumgartner,  1979,  pp.  3-4). 

Specialists  vs.  generalists.  The  Commission  found  that  the  military’s 
preference  for  generalists  in  the  uniformed  ranks  was  inconsistent  with  the 
specialization  needed  to  address  the  increasing  complexity  of  defense  management, 
especially  since  many  key  management  positions  were  filled  by  uniformed  personnel 
(MacNeil  &  Metz,  1956,  pp.  268-269).  In  particular,  the  Commission  cited  the 
military  Services’  practice  of  frequently  rotating  officers  in  and  out  of  administrative 
and  operational  positions  as  detrimental  to  efficient  management  (p.  246).  It 
recommended  that,  with  necessary  Congressional  approval,  the  DoD  develop  a 
personnel  system  that  would  provide  career-trained  managers  for  functional  areas 
such  as  acquisition.  Such  a  system  would  address  specialized  requirements  for 
training,  assignments,  rotations,  and  promotions,  both  for  uniformed  and  civilian 
personnel  (pp.  269-270).  Such  recommendations  promote  the  view  of  the  PM  as  a 
professional  member  of  a  specialized  career  field. 

Civilian  vs.  military  control.  The  Commission  remarked  on  military  officers’ 
general  “anti-economic  and  unbusinesslike”  attitude  that,  understandably,  favors 
getting  as  many  resources  necessary  to  do  their  job  (i.e. ,  win  armed  conflicts),  as 
opposed  to  getting  along  with  as  little  as  possible  (MacNeil  &  Metz,  1956,  p.  198).  It 
opined  that  “[t]he  management  aspect  of  national  defense  ...  requires  specialized 
skills  and  attitudes  normally  found  among  civilians”  (p.  273).  While  recognizing  the 
benefits  of  a  military  officer’s  field  experience  in  understanding  the  requirements  and 
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operational  employment  of  proposed  weapons  system  (also  recognized  in  many  of 
the  studies  cited  in  subsequent  sections),  the  Commission  advocated  strengthening 
and  increasing  civilian  control  of  military  administrative  functions  and  increased 
application  of  business  methods,  particularly  in  supply-related  activities  (MacNeil  & 
Metz,  1956,  pp.  268-269).  These  positions  presaged  policy  initiatives  in  later  years 
to  increase  the  numbers  of  civilians  in  PM  positions  and  to  emphasize  business 
skills  in  PM  training. 

B.  Early  DoD  Professionalism  Actions 

In  1965,  the  DoD  published  its  first  regulations  on  the  required  qualifications, 
training,  and  tenure  durations  for  acquisition  PMs.  These  were  revised  in  1971  and 
1974,  and  again  in  1986,  to  reflect  evolving  legislative  requirements. 

In  1964,  the  DoD  established  the  Defense  Weapon  Systems  Management 
Center  at  Dayton,  OH,  for  the  purpose  of  training  defense  acquisition  PMs,  mainly 
via  a  10-week  Project  Management  Course.  It  was  replaced  in  1971  by  the  Defense 
Systems  Management  School  (later  College  [DSMC])  at  Fort  Belvoir,  VA,  which  was 
intended  to  provide  in-depth  knowledge  and  preparation  on  advanced  management 
concepts  to  prospective  PMs  (Layton,  2007,  pp.  6-7).  PM  training  continues  at  Fort 
Belvoir  to  the  present  under  the  Defense  Acquisition  University  (DAU). 

From  the  late  1960s  through  the  1980s,  each  of  the  military  Services  also 
established  its  own  programs  to  train  PMs,  for  example,  the  Army’s  PM 
Development  Program  (Baumgartner,  1979,  p.  100)  and  the  Navy’s  Weapon  System 
Acquisition  Management  and  Materiel  Professional  programs  (Fox,  1988,  p.  201). 

C.  Commissions  during  the  1 970s 

The  Fitzhuqh  Commission.  The  1970  report  of  President  Nixon’s  Blue  Ribbon 
Defense  Panel,  chaired  by  Gilbert  Fitzhugh,  presented  the  first  in-depth  examination 
of  weapons  acquisition  as  part  of  its  comprehensive  review  of  national  defense 
management  (McKinney,  Gholz,  &  Sapolsky,  1994).  It  noted  deficiencies  similar  to 
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those  found  by  the  Hoover  Commission.  Specifically,  the  Fitzhugh  Commission 
found  that  the  personnel  systems  of  the  military  Services  did  “not  facilitate  career 
development  in  the  technical  and  professional  activities,  such  as  research  and 
development  [and]  procurement”  (Blue  Ribbon  Defense  Panel,  1970,  p.  2). 
Regarding  PMs,  it  found  “[no]  indication  of  consistent  efforts  ...  to  select  Program 
Managers  from  among  those  officers  who  have  the  most  promising  potential.  ...  [A 
PM]  should  possess  both  managerial  and  technical  skills  and  experience  in  the 
operational  employment”  of  the  system  under  development  (p.  80).  It  made 
extensive  recommendations  for  improvement,  including  several  focused  on  the  PM, 
as  follows: 

■  Establishing  career  specialty  codes  for  military  PMs,  along  with 
selection  and  training  criteria  to  ensure  an  adequate  supply  of  qualified 
officers; 

■  Increasing  the  number  of  qualified  civilian  PMs;  and 

■  Increasing  the  duration  of  PM  assignments  and  allowing  for  overlap  of 
incumbent  and  replacement  PM  assignments  to  enhance  program 
continuity  and  stability  (Lockwood,  1985,  p.  2). 

The  Commission  on  Government  Procurement.  Reporting  two  years  after  the 
Fitzhugh  Commission,  this  Congressionally  chartered  commission  addressed  the 
same  general  issues  of  procurement  problems  and  reform.  It  had  few 
recommendations  regarding  the  PM  professionalism  issue,  however,  other  than  to 
increase  the  number  of  civilians  as  major  PMs  as  a  means  to  increase  managerial 
expertise  and  reduce  turnover  in  that  position  (McKinney,  Gholz,  &  Sapolsky,  1994, 
pp.  5,  13). 

D.  Congressional  Interest  and  Actions  during  the  ’70s  and 
’80s 

As  early  as  1954,  Congress  had  expressed  concern  that  frequent  rotations 
and  short  assignment  durations  among  PMs  were  “disturbing  and  harmful  to  the 
productivity”  of  acquisition  programs  (in  Fox,  1988,  p.  178).  These  concerns 
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resurfaced  occasionally,  for  example,  in  testimonies  like  those  of  the  1970  Fitzhugh 
Commission,  of  Deputy  Secretary  of  Defense  Packard  in  1971  (Fox,  1974,  p.  200), 
and  of  Admiral  Rickover  in  1971  when  commenting  on  the  inexperience,  lack  of 
skills,  and  short  assignments  of  PMs  in  the  DoD  (Lockwood,  1985,  pp.  1 1-12).  The 
PM  professionalism  issue  was  typically  presented  in  the  context  of  concerns  with 
acquisition  management  in  general,  like  the  GAO’s  (1979)  report  on  government 
procurement  which  included  criticisms  of  the  lack  of  qualifications  among  DoD  PMs. 

Former  Congressional  staff  members4  Colleen  Preston  (2011)  and  Jon 
Etherton  (2011)  both  note  that  Congressional  attention  to  PM  tenure  reflected  its 
concerns  with  accountability,  since  frequent  rotations  made  it  difficult  to  hold  a  PM 
accountable  for  program  shortcomings.  As  Preston  put  it,  “[W]hen  program 
managers  came  in  to  testify  before  members  of  the  committee  about  their  program, 
we  got  a  lot  of,  ‘It  didn’t  happen  on  my  watch’”  (2011).  Thus,  lengthening  PM  tenure, 
whether  by  using  civilian  PMs  to  a  greater  extent  or  by  mandating  minimum  tenure 
lengths,  would  serve  the  dual  purposes  of  providing  managerial  stability  to  enhance 
program  outcomes  and,  in  the  case  of  poor  outcomes,  helping  to  ensure  that  the 
responsible  PM  could  be  held  accountable  for  those  outcomes. 

Exacerbating  the  tenure  problem  was  the  perception  in  the  DoD  that 
acquisition  was  not  a  particularly  attractive  or  career  enhancing  field.  One  issue  was 
that  PMs  were  perceived  to  have,  as  Fox  put  it,  “minimal  authority  [with]  maximum 
responsibility”  (1984,  p.  156).  That  is,  they  had  nominal  charge  over  their  programs, 
but  with  little  real  authority  because  of  the  many  overseers  and  stakeholders  who 
held  sway  over  their  programs’  direction  (Etherton,  2011).  Further,  acquisition  didn’t 
appear  to  have  the  same  level  of  promotion  opportunities  as  operational  career 
fields.  Gansler  (201 1  b)  has  noted  the  absence  of  acquisition  professionals  in  the 
most  senior  flag  officer  ranks  of  the  DoD  during  these  years.  Regarding  the  four-star 


4  Preston  served  as  General  Counsel  to  the  Investigations  Subcommittee  of  the  House  Armed 
Services  Committee;  Etherton  served  on  the  professional  staff  of  the  Senate  Armed  Services 
Committee’s  Subcommittee  on  Acquisition  and  Technology. 
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general  commander  of  Army  Materiel  Command  (the  Army’s  principal  acquisition 
agency),  four  consecutive  commanders  had  no  acquisition  experience,  which  sent 
clear  signals  to  junior  acquisition  professionals  about  their  career  field:  “[l]f  you  look 
up  and  see  that  the  people  above  you  don’t  have  any  acquisition  experience,  you 
know  that  [experience]  is  not  going  to  be  emphasized.  Career  planning  for 
acquisition  wasn’t  really  important”  (Gansler  201 1b).  Hence  the  view  among 
reformers  of  the  importance  of  structural  changes  to  the  DoD’s  acquisition  career 
management  systems. 

In  1984,  two  years  after  President  Reagan  signed  an  executive  order 
directing  the  establishment  of  career  management  programs  for  professional  federal 
employees,  a  Senate  Armed  Services  subcommittee  began  the  first  series  of 
Congressional  hearings  targeted  specifically  on  the  issue  of  PM  professionalism. 
These  hearings  coincided  roughly  with  passage  of  the  first  legislative  provisions  in 
1984  and  1985  (Department  of  Defense  Authorization  Act  for  1985;  Department  of 
Defense  Authorization  Act  for  1986).  This  legislation  directed  the  lengthening  of  PM 
assignment  durations  to  four  years  or  the  completion  of  a  major  program  milestone, 
and  for  PMs,  requirements  for  minimum  training  (mandatory  attendance  at  DSMC) 
and  experience  (a  minimum  of  eight  years  in  acquisition-related  assignments,  two  of 
which  must  be  with  major  systems  acquisition;  Lockwood,  1985,  pp.  3-4,  13,  15). 

According  to  Etherton  (2011),  then-Senator  Dan  Quayle,  chairman  of  the 
Defense  Acquisition  Policy  Subcommittee,  took  a  different  tack  on  the  PM 
professionalism  issue  in  the  mid-1980s.  After  meeting  with  several  PMs  (and 
hearing  the  “responsibility  without  authority”  complaint  referred  to  in  this  section), 
Quayle  decided  to  focus  on  empowering  them,  an  initiative  that  led  to  creation  of  the 
Defense  Enterprise  Program  (DEP;  enacted  in  the  National  Defense  Authorization 
Act  for  1987  [1986]).  Something  of  a  managerial  experiment,  the  DEP  authorized 
the  DoD  to  nominate  selected  programs  that  would  have  reduced  regulatory  burdens 
and  reporting  requirements.  Ideally,  this  would  allow  those  PMs  to  have  greater 
flexibility  in  exercising  good  managerial  skills  and  discretion,  while  at  the  same  time 
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improving  PM  accountability.  The  DEP  encountered  resistance  within  the  DoD, 
however,  and  as  Congressional  leadership  changed  over  the  next  few  years,  it 
proved  to  be  short-lived. 

In  1986,  the  GAO  issued  a  detailed  report  on  the  acquisition  workforce, 
concluding  that  DoD  PMs  in  general  lacked  the  appropriate  training,  education,  and 
experience  needed  for  the  challenges  of  managing  acquisition  programs  (GAO, 

1986). 

Throughout  the  1980s,  PM  tenure  remained  the  principal  and  most  intractable 
professionalism  reform  issue  for  Congress  (Etherton,  2011).  One  reason  was  the 
prominence  of  the  personnel  subcommittees  of  the  Armed  Services  Committees  in 
Congress  which  “ruled  the  roost”  (Etherton,  201 1 )  on  any  military  personnel  matters. 
Another  reason  was  competing  priorities  about  how  to  achieve  lengthened  tenure, 
whether  through,  for  example,  mandates  for  increased  numbers  of  civilian  PMs,  or 
for  establishment  of  dedicated  career  fields  that  would  allow  military  PMs  to  serve  for 
longer  durations. 

E.  Executive  Branch  Actions  during  the  1980s 

The  Carlucci  Initiatives.  Only  a  few  months  after  President  Reagan’s 
inauguration  in  1981,  Deputy  Secretary  of  Defense  Frank  Carlucci  introduced  a  set 
of  over  30  diverse  measures  (subsequently  known  variously  as  the  Defense 
Acquisition  Improvement  Program  [DAIP],  or  more  informally,  the  Carlucci  Initiatives) 
intended  to  improve  acquisition  outcomes.  Among  the  initiatives  was  “Increase 
program  stability”  through  measures  such  as  increased  PM  tenure.  Although  some 
of  the  initiatives  resulted  in  limited  progress,  by  1984  the  DoD’s  commitment  to  the 
DAIP  had  largely  dissipated  (Munechika,  1997,  pp.  6-9). 

The  Grace  Commission.  President  Reagan’s  Private  Sector  Survey  on  Cost 
Control,  the  Grace  Commission,  addressed  the  entire  scope  of  government  operations 
to  address  waste  in  spending,  as  its  name  suggests.  While  making  many  acquisition- 
related  recommendations,  the  Commission’s  recommendations  regarding  PM 
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professionalism  generally  repeated  prior  calls  for  better  trained  and  more  experienced 
PMs  (Grace  Commission,  1984). 

The  Packard  Commission.  In  late  1985,  President  Reagan  established  the 
Blue  Ribbon  Commission  on  Defense  Management,  chaired  by  former  Deputy 
Defense  Secretary  David  Packard,  in  response  to  the  rash  of  acquisition  “horror 
stories”  during  the  mid-1980s  (Blue  Ribbon  Commission,  1986,  p.  1).  Because  its 
major  task  was  to  evaluate  the  entire  U.S.  defense  acquisition  system,  the 
Commission  also  commented  on  the  professionalism  issue.  Assuming  the  central 
importance  of  the  PM  position  (Berteau,  2011)  and  the  necessity  of  experience  for 
success  (Gansler,  2011b),  the  Commission  found  that,  compared  to  that  of  private 
industry,  the  defense  acquisition  workforce  is  “undertrained,  underpaid,  and 
inexperienced”  (p.  28).  While  supporting  the  legislation  cited  previously,  it  affirmed 
the  importance  of  further  improvements  to  PM  training  and  career  development,  both 
for  military  and  civilian  members. 

Secretary  of  Defense  Cheney’s  1989  Defense  Management  Review  generally 
echoed  the  Packard  Commission’s  findings  on  the  state  of  the  acquisition  workforce 
(Mavroules,  1991,  p.  18).  Because  recommendations  for  improvement  efforts  were 
developed  by  the  individual  military  Services,  implementation  plans  varied 
significantly  across  the  DoD  (Layton,  2007,  pp.  10-11). 

F.  The  1990  House  Armed  Services  Committee  (HASC) 

Report 

By  the  late  1980s,  interest  in  PM  professionalism  in  the  context  of  continuing 
poor  acquisition  outcomes  had  grown  to  the  point  that  Congress  was  considering 
major  reform  legislation  (Etherton,  2011;  Preston,  2011).  According  to  Rep. 
Mavroules,  Congress  reflected  a  consensus  that  most  previous  efforts  to  reform  the 
acquisition  system  had  focused  on  processes  and  structures,  and  that  what  was 
presently  needed  was  a  focus  on  people.  This  consensus,  however,  was  based 
mainly  on  anecdotal  evidence  (Preston,  201 1).  Thus,  as  a  necessary  prelude  to 
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workforce  reform  legislation,  the  House  Armed  Services  Committee  (HASC) 
chartered  an  “in-depth  analysis  of  the  state  of  the  acquisition  workforce”  (HASC, 

1 990,  p.  1 )  to  focus  on  the  following  four  questions  (p.  1 ): 

1.  Are  acquisition  managers,  including  PMs,  being  appointed  with  the 
required  qualifications,  and  are  they  remaining  in  their  positions  for  the 
minimum  four-year  duration? 

2.  Is  there  a  career  program  to  develop  professional  and  qualified 
personnel,  both  military  and  civilian? 

3.  Does  the  workforce  reflect  an  appropriate  mix  of  military  and  civilian 
personnel? 

4.  What  impediments  exist  to  workforce  professionalism,  and  how  may 
those  be  overcome? 

The  776-page  HASC  report  provided  detailed  and  comprehensive  data  on 
almost  all  facets  of  the  workforce’s  qualifications,  providing  unprecedented  insights 
and  a  much-needed  empirical  basis  for  subsequent  legislation.  The  report’s  findings 
generally  supported  the  momentum  in  Congress  for  major  workforce  legislation.  For 
example,  it  found  the  following  (Mavroules,  1991,  pp.  17-21): 

1 .  Fewer  than  half  of  major  PM  positions  were  filled  with  graduates  of  the 
statutorily  required  PM  course  at  Fort  Belvoir. 

2.  Since  1984,  only  six  of  94  major  PMs  had  remained  in  their 
assignments  for  the  statutory  four-year  (or  major  milestone  completion) 
requirement. 

3.  The  DoD’s  career  programs  for  PMs  lacked  key  elements  leading  to 
professionalism. 

4.  While  civilians  made  up  over  90%  of  the  total  acquisition  workforce, 
only  about  10%  were  assigned  to  major  PM  positions. 

The  report  also  found  differing  levels  of  attention  to  PM  professionalism 
among  the  three  military  Services.  For  example,  only  the  Air  Force  had  a  dedicated 
professional  career  development  and  certification  program  for  military  PMs. 
Accordingly,  the  Air  Force  had  more  experienced  PMs  (with  an  average  of  17  years 
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experience)  than  the  Army  or  Navy,  as  well  as  a  higher  percentage  (97%)  of  PMs 
with  at  least  eight  years  of  acquisition  experience  than  either  the  Army  (81%)  or  the 
Navy  (71%).  Regarding  PM  tenure,  the  least  experienced  Navy  PMs  had  an 
average  tenure  of  41  months,  while  the  more  experienced  Air  Force  PMs  averaged 
only  25  months;  Army  PMs  averaged  29  months  (HASC,  1990,  pp.  32-45). 

G.  Differences  in  PM  Career  Management  Among  the  Military 
Departments 

Differences  in  the  career  management  approaches  of  each  of  the  military 
departments  had  been  evident  prior  to  the  HASC  report.  Preston  (2011)  and 
Etherton  (2011)  both  describe  the  Air  Force  as  having  the  most  well-developed 
career  management  policies  and  procedures  for  PMs.  As  noted  previously,  the  Air 
Force  had  a  dedicated  career  path  and  was,  as  Preston  put  it,  “growing  true 
business  managers  from  the  very  outset”  (201 1 ).  Etherton  (201 1 )  explained  the  Air 
Force’s  priority  on  PMs  as  a  consequence  of  its  higher  capital-labor  ratios  than 
either  of  the  other  Services. 

The  Army’s  PM  career  program  envisioned  accessing  mid-level  officers  from 
the  operational  ranks  and  providing  developmental  assignments  prior  to  their 
selection  for  PM  by  a  centralized  selection  board.  While  the  Army’s  program  was 
not  as  well  developed  as  the  Air  Force’s,  Baumgartner  (1979,  pp.  99-103)  wrote  in 
glowing  terms  of  Army  PMs  and  their  professional  qualifications,  referring  to  them  as 
“princes  of  the  realm.” 

The  Navy  was  recognized  as  having  the  least  developed  PM  career  program. 
Preston  recalls, 

[T]hey  were  pulling  people  right  off  a  ship,  rotating  through  a  short  tour, 
turning  them  into  program  managers,  and  thinking  that  if  they  sent  them  to 
DSMC  ...  that  was  going  to  be  enough.  ...  [T]hey  would  be  capable  of  making 
. . .  multi-million  dollar  business  decisions.  (2011) 
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One  interviewee,  a  former  Navy  PM  who  chose  to  remain  anonymous, 
attributed  the  Navy’s  approach  to  its  historical  preference  that  its  PMs  be  proven 
leaders  with  recent  operational  experience  in  the  program’s  domain  (e.g.,  a 
submariner  as  PM  of  a  submarine  program).  The  Navy  saw  career  Air  Force  PMs 
as  essentially  “civilians  wearing  uniforms,”  because  they  had  little  to  no  operational 
experience.  Business  expertise  and  program  continuity,  in  the  Navy’s  model,  were 
best  provided  by  civilian  deputies  to  the  uniformed  PMs. 

As  might  be  expected,  these  career  management  differences  among  the 
three  departments  complicated  the  task  of  PM  reform  efforts  by  both  Congress  and 
the  DoD.  Resistance  was  particularly  strong  from  the  Air  Force,  which — perhaps 
with  some  justification — felt  that  its  program  was  already  strong,  and  that  it  did  not 
need  “interference”  from  outside  (Preston,  2011). 

Etherton  (201 1 )  noted  that  the  military  departments’  PM  career  programs 
were  focused  only  on  uniformed  members: 

Whether  [career  management  for  military  PMs]  was  adequate  was  another 
question,  but  on  the  civilian  side  there  was  no  equivalence.  ...  If  you  were  a 
civilian  you  didn’t  have  the  same  career  opportunities  as  your  military 
counterpart.  [Congressional  reformers  wanted]  to  create  room  for  civilian 
program  managers  to  operate  more  effectively  in  the  system  and  have  more 
opportunities. 

This  new  focus  on  civilians  was  related  to  the  goal  of  making  acquisition  an 
attractive  professional  career  field,  as  described  by  Etherton  (201 1 ): 

...so  that  someone  coming  in  from  the  outside  who  was  a  talented  manager 
or  had  some  interest  in  public  service  would  look  at  the  Acquisition  Corps  as 
something  to  be  sought  after — to  make  it  more  attractive,  to  make  it 
something  that  would  have  special  standing. 

H.  The  Defense  Acquisition  Workforce  Improvement  Act 
(DAWIA) 

Enacted  in  November  1990  as  part  of  the  1991  Defense  Authorization  Act, 
the  DAWIA  provided  for  a  broad  range  of  workforce  professionalization  actions,  not 
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only  for  PMs,  but  for  other  acquisition  specialists  as  well.  It  required  the  Secretary  of 
Defense  to  establish  policies  and  procedures  for  acquisition  career  management 
and  to  ensure  their  uniformity  throughout  the  DoD.  Some  of  its  main  provisions 
regarding  PMs  included  the  following: 

■  Establishment  of  designated  career  fields,  career  managers,  career 
advisory  boards,  and  career  paths  for  advancement; 

■  Establishment  of  an  Acquisition  Corps  with  certification  levels  in  each 
acquisition  career  field,  along  with  training,  education,  and  experience 
requirements  necessary  for  certification  at  each  level; 

■  Limitations  on  preferences  for  military  personnel  and  requirements  to 
increase  numbers  of  civilian  PMs; 

■  Establishment  of  the  Defense  Acquisition  University  (DAU)  for  the 
professional  education  and  development  of  the  workforce; 

■  Reinforcement  of  the  four-year/major  milestone  tenure  requirement  for 
major  PMs;  and 

■  Reinforcement  of  the  requirement  to  attend  the  PM  course  prior  to 
assignment,  along  with  experience  requirements:  for  a  major  PM,  eight 
years  of  acquisition  experience,  two  of  which  must  be  in  a  major 
program  office;  for  a  nonmajor  PM,  six  years  of  acquisition  experience. 

Waivers  could  be  granted  for  these  last  two  requirements. 

Etherton  (2011)  noted  that  policy-makers  intended  that  the  DAWIA  (1990) 
and  other  reforms,  such  as  the  establishment  of  the  program  executive  office  (PEO) 
structure  (Horgan,  1995),  be  mutually  supportive  regarding  enhanced  PM 
professionalism  and  accountability.  It  was  presumed  that  removing  managerial 
layers  between  the  PM  and  milestone  decisions  authorities,  along  with  establishing 
clearer  lines  of  reporting  and  authority,  would  enable  better  decision  making  by 
better  trained  and  more  experienced  PMs. 

Compared  with  the  years  leading  up  to  the  DAWIA  (1990),  relatively  little 
policy  attention  was  paid  to  the  PM  professionalization  issue  following  its  passage 
through  the  early  2000s.  This  can  be  attributed  to  perceptions  that  the  DAWIA’s 
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sweeping  reforms  were  for  the  most  part  adequate  and  that  time  was  needed  for 
their  proper  implementation.  It  also  reflected  the  DoD’s  concern  with  larger  issues  of 
strategic  human  capital  planning  (GAO,  2002),  which  was  sorely  needed  after  the 
major  acquisition  workforce  reductions  during  the  1990s. 

One  unintended  consequence  of  the  DAWIA  (1990)  was  described  in  an 
anecdote  related  by  a  senior  DoD  acquisition  official  who  chose  to  remain 
anonymous.  At  a  meeting  during  the  mid-1990s,  the  official  inquired  as  to  the  quality 
of  acquisition  personnel  compared  to  ten  years  previously.  A  3-star  flag  officer 
replied,  “Of  course,  it’s  lower.  ...  I  have  36  0-6s5  coming  up  [for  assignment]  this 
year,  and  I  have  42  jobs.  ...  There  was  a  time  when  I  had  42  0-6s  and  31  jobs.  You 
could  weed  [the  weaker  performers]  out;  we’re  not  able  to  weed  anymore.”  Thus, 
the  DAWIA’s  certification  and  assignment  standards  appear  to  have  had  the  effect  of 
reducing  the  pool  of  available  senior  officers  to  the  point  that,  at  least  according  to 
this  anecdote,  “best  qualified”  criteria  for  selection  to  PM  were  rendered  essentially 
meaningless. 

I.  Post-DAWIA  Interest  in  PM  Professionalization 

Roughly  ten  years  after  the  DAWIA’s  (1990)  passage,  major  PMs  resurfaced 
as  an  issue  of  interest,  mainly  regarding  the  problem  of  brief  tenures.  For  example, 
an  unpublished  report  in  2001  provided  data  on  major  PM  tenure  since  passage  of 
the  DAWIA,  noting  that  on  average,  DoD  PMs  had  failed  to  meet  its  statutory 
requirements  (Office  of  the  Director,  2001). 

In  2005,  the  Defense  Acquisition  Performance  Assessment  (DAPA)  Project 
called  for  the  PM  tenure  requirement  to  be  established  at  the  point  of  formal  program 
initiation  and  proceed  through  completion  of  operational  tests  in  order  to  enhance 
leadership  stability  (Assessment  Panel,  2005).  In  2008,  the  GAO  reported  once 
again  on  the  issue  of  short  PM  tenure  durations,  stating  that  frequent  program 


5  Colonels  in  the  Air  Force  and  Army;  Captains  in  the  Navy 
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manager  turnover  has  “potential  to  impact  acquisition  outcomes  on  individual 
programs”  (p.  2).  The  GAO  noted: 

[F]or  39  major  acquisition  programs  started  since  March  2001 ,  the  average 
time  in  system  development  was  about  37  months.  The  average  tenure  for 
program  managers  on  those  programs  during  that  time  was  about  17 
months — less  than  half  of  what  is  required  by  DoD  policy,  (p.  11) 

While  not  attributing  adverse  program  outcomes  to  this  tenure  issue,  the  GAO 
concluded  that  outcomes  in  terms  of  cost,  schedule,  and  performance  had  not 
improved  since  2000. 

The  National  Defense  Authorization  Act  for  2007  (2006).  Provisions  of  this 
legislation  put  a  different  emphasis  on  major  acquisition  PMs.  Congress  called  for 
the  DoD  to  increase  PM  empowerment  and  responsibility  through  revised  guidance 
on  their  qualifications,  resources,  tenure,  and  accountability. 

A  year  later,  in  reporting  to  Congress  on  its  progress  toward  meeting  these 
requirements,  the  DoD  described  acquisition  as  consisting  of  two  related 
environments  (2007,  pp.  6-9).  “Big  A”  acquisition  is  the  larger  DoD  institutional 
environment  that,  in  addition  to  strategic  acquisition  management,  also  entails 
requirements  management  and  resource  management,  two  activities  that  are 
typically  outside  a  PM’s  sphere  of  influence.  “Little  a”  acquisition  is  the  managerial 
environment  of  activities  within  a  PM’s  ability  to  influence.  The  DoD  stated  that  even 
with  Congressionally  directed  PM  enhancements  that  might  enable  improvements  in 
little  a  acquisition,  improved  outcomes  were  not  likely  without  improvements  to  the 
Big  A  components  of  the  acquisition  system  (2007,  p.  8)  the  GAO  rendered  a  similar 
opinion  only  three  months  later  (2007,  p.  7).  This  represented  a  substantive  step  back 
by  DoD’s  acquisition  leadership  from  the  conventional  wisdom  regarding  PM 
professionalism  and  program  outcomes. 

Business  Executives  for  National  Security  (BENS).  This  group,  consisting  of 
several  industry  leaders  and  former  senior  government  officials  (e.g.,  Norm 
Augustine,  Gary  Hart,  Warren  Rudman),  published  its  report  in  2009  on 
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recommendations  for  reforming  defense  acquisition.  Like  other  commissions 
mentioned  previously,  it  addressed  a  wide  range  of  issues,  including  the  acquisition 
workforce  and  major  PMs,  but  it  naturally  reflected  industry’s  perspectives.  Pertinent 
recommendations  included  strengthening  the  role  of  industry  in  workforce  training 
(BENS,  2009,  p.  32),  implementing  private-sector  organizational  forms  and 
management  practices,  improving  PM  training  and  experience,  and  lengthening 
durations  of  PM  tenure  (p.  34).  Despite  all  of  the  professionalizing  actions  taken  by 
the  DoD  since  the  DAWIA  (1990),  the  report  noted,  “the  acquisition  career  field  is  not 
viewed  as  a  profession  in  the  same  sense  as  are  operational  billets”  (BENS,  2009, 
p.  35). 


Defense  Business  Board  (DBB).  In  December  2010,  Under  Secretary  of 
Defense  Carter  chartered  the  DBB  to  conduct  a  review  of  the  DoD’s  PMs  and  make 
recommendations  for  improving  their  effectiveness  based  on  private  sector  best 
practices.  The  DBB  conducted  its  review  and  provided  its  report  four  months  later 
after  interviews  with  OSD  staff,  acquisition  executives,  six  PMs,  and  representatives 
of  eight  large  firms  (DBB,  201 1 ,  pp.  1-2).  It  found  “compelling  reasons”  to  change 
the  way  the  DoD  manages  its  PMs  (p.  2),  specifically  regarding  the  prevalence  of 
short  assignment  durations  that  “negatively  affect”  acquisition  programs  (p.  2). 

According  to  the  DBB,  industry  perceived  that  the  DoD’s  acquisition 
performance  is  complicated  by  absences  and  turnover  of  military  PMs  who  tend  to 
be  in  charge  of  major  programs.  It  reported  that,  while  private  sector  PMs  enjoyed 
“increased  PM  tenure,  continuity,  and  business  acumen,”  the  DoD’s  military  PMs 
tended  to  be  “risk  adverse  [sic]”  due  to  their  inexperience  and  excessive  oversight 
requirements  (pp.  2-3).  The  DBB  offered  two  choices  for  reform:  Professionalize 
PMs  (in  the  sense  of  creating  a  “career  destination”  for  them,  as  in  a  true  profession 
such  as  law  or  medicine),  or  reserve  major  PM  positions  for  civilians  who  can  serve 
for  longer  durations. 
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J.  Summary  of  Policy  Review 

This  admittedly  lengthy  review  is  provided  to  make  the  following  points:  (1) 
the  issue  of  PM  professionalism  has  been  a  feature  of  defense  acquisition  reform 
policy  for  over  fifty  years;  (2)  numerous  policy  initiatives  during  those  years  have 
attempted  to  make  PMs  professional;  and  (3)  because  the  PM  issue  persists,  those 
attempts  have  provided  less  than  satisfactory  results.  Notably  absent  from  this 
review,  however,  is  any  policy  analysis  that  attempts  to  establish  a  relationship 
between  enhanced  professionalism  and  improved  program  outcomes. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-32- 


Literature  Review 


Because  little  research  directly  addresses  the  relationship  between  PM 
professionalism  and  program  outcomes,  this  section  also  summarizes  some  relevant 
research  from  the  field  of  human  resource  management. 

A.  Early  Studies 

Scholarly  attention  to  the  PM  coincides  roughly  with  the  rise  of  modern  project 
management  in  the  mid-20th  century  with  highly  complex  weapons  projects  such  as 
the  Manhattan  Project  and  the  Atlas  project  (Kwak,  2005).  The  leadership  and  skills 
of  the  two  generals  who  managed  these  projects — Leslie  Groves  and  Bernard 
Schriever,  respectively —  have  been  widely  cited,  reinforcing  the  view  of  the  PM’s 
importance  in  achieving  successful  results  (Baumgartner,  1963,  1979).  Gaddis’ 
Harvard  Business  Review  article  (1959)  described  the  unique  roles  and 
competencies  of  this  new  type  of  manager.  Peck  and  Scherer’s  (1962)  seminal 
study  of  defense  acquisition  noted  the  importance  of  PM  experience  and  the 
disruption  to  programs  that  occur  with  frequent  PM  rotations  (pp.  92-93).  The 
influential  works  of  Katz  (1974)  and  Mintzberg  (1975)  documented  the  attributes  and 
competencies  of  effective  managers. 

B.  Fox’s  Studies 

J.  Ronald  Fox’s  two  books  (1974,  1988)  remain  the  two  most  influential  and 
comprehensive  analyses  of  U.S.  defense  acquisition  management.  In  each,  Fox 
devotes  an  entire  chapter  to  the  PM,  with  emphasis  on  the  professionalism  issue, 
covering  the  material  summarized  in  the  previous  chapter  in  much  greater  detail. 
While  lauding  the  efforts  of  individual  PMs  to  do  their  best  in  a  flawed  system,  Fox 
presents  trenchant  critiques  of  an  acquisition  culture  that  steadfastly  resists  reform, 
including  PM  professionalism  efforts. 
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In  these  books,  Fox  makes  claims  regarding  the  PM  issue  that  are  relevant 
for  this  present  study.  For  example,  he  states,  “The  capability  of  the  program 
manager  and  his  staff  obviously  determines  the  ultimate  success  of  each  weapon 
acquisition  program”  (1974,  p.  180).  Similarly,  he  cites  “the  reality  that  an  effective 
and  efficient  acquisition  program  requires  advanced  program  management  skills, 
based  on  extensive  practical  training  and  years  of  program  management 
experience”  (1988,  p.  196).  Unfortunately,  he  offers  no  support  for  such  claims. 
Thus,  while  he  does  give  compelling  evidence  both  for  the  lack  of  PM 
professionalism  and  for  unfavorable  program  outcomes,  he  simply  assumes  the 
former  contributes  to  the  latter.  Considering  the  widespread  influence  of  Fox’s 
works,  it  is  unsurprising  that  many  apparently  accept  the  “reality”  of  this  relationship 
(see,  for  example,  Blue  Ribbon  Commission,  1986,  pp.  27-28;  Lockwood,  1985,  p. 
1). 

C.  Project  Management  Studies 

Project  management  literature  contributes  to  the  present  study  in  several 
ways.  First,  the  question  of  defining  and  measuring  project  success,  especially  for 
complex  projects  with  diverse  stakeholders,  generates  much  attention  (see,  for 
example,  Pinto  &  Slevin,  1988).  Agreement  on  specific  success  criteria  is  elusive, 
and  most  agree  on  only  general  standards  like  overall  mission  accomplishment, 
stakeholder  satisfaction,  and  conformance  to  quality,  cost,  and  schedule  targets 
(Baker,  Murphy,  &  Fisher,  1988;  Murphy,  Baker,  &  Fisher,  1974).  This  problem  is 
exacerbated  by  the  lack  of  data  on  achievement  of  cost,  schedule,  and  quality 
targets,  whether  because  of  proprietary  concerns  (mainly  in  the  private  sector)  or 
simply  because  data  have  not  been  systematically  collected.  As  a  result,  most 
studies  rely  on  stakeholders’  subjective  assessments  to  determine  program 
outcomes  (Crawford,  2002). 

Several  of  these  studies  have  attempted  to  quantify  the  benefits  of  project 
management  and  project  management  techniques  in  terms  of  favorable  project 
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outcomes  in  the  private  sector  (see,  for  example,  Ibbs  &  Reginato,  2002;  Morris, 
2002).  None,  however,  has  focused  on  the  PM  or  on  defense  acquisition  projects. 


Numerous  studies  have  focused  on  the  issue  of  desired  PM  competencies 
(Geoghegan  &  Dulewicz,  2008;  Posner,  1987;  Thamhain,  1991),  to  include  defense 
acquisition  PMs  (Cullen  &  Gadeken,  1990;  Gadeken,  1994,  2002;  McVeigh,  1994). 
All  of  these  studies,  however,  relied  on  subjective  assessments,  both  for  success 
factors  and  for  program  outcomes.  Because  they  examined  competencies  (e.g., 
leadership  skills),  these  studies  did  not  address  any  of  the  criteria,  such  as  training 
and  tenure,  that  received  attention  in  prior  critiques  of  PM  professionalism. 

Like  Fox,  then,  PM  scholars  have  assumed,  but  failed  to  provide  evidence  of, 
a  relationship  between  PM  competencies  or  attributes  and  the  outcomes  of  the 
programs  they  lead. 

D.  Human  Resource  Management  (HRM)  Literature 

The  difficulty  in  identifying  relationships  between  PM  professionalism  and 
program  outcomes  is  illustrated  by  human  resource  management  (HRM)  studies  that 
address  the  issue  of  “return  on  investment”  (ROI)  from  training,  education,  and  other 
development  programs  (Dionne,  1996;  Kirkpatrick,  1994;  Rowden,  2005).  As  Wilson 
(2004)  notes,  many  organizations  take  a  “leap  of  faith”  when  it  comes  to  HR 
development  programs,  presuming  their  inherently  positive  effects  for  all  concerned, 
including  for  the  bottom  line.  ROI  studies  attempt  to  replace  faith  with  evidence, 
whether  motivated  by  executive  concerns  with  profitability  (Bartel,  2000;  Phillips, 
1996)  or  by  HR  specialists’  concerns  to  justify  their  budgets  (Geber,  1996;  Institute 
of  Management  &  Administration,  2003). 

Despite  occasional  claims  to  the  contrary  (e.g.,  Phillips,  1997),  most  scholars 
find  significant  difficulties  in  establishing  relationships  between  HR  development 
efforts  and  effects  such  as  increased  profitability  for  any  other  than  relatively 
formulaic  or  routine  tasks  (see,  for  example,  Mikesell,  Wilson,  &  Lawther’s  [1975] 
study  of  training  for  tax  auditors;  also,  Brown  &  Seidner,  1998;  Stolovitch  &  Maurice, 
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1998;  Swanson,  1998).  For  managerial  and  executive  development  efforts,  issues 
of  training  validity  (treatment  effects  from  the  program)  and  performance  validity 
(transferring  performance  from  the  program  to  the  job)  present  major  challenges 
(Goldstein,  1979).  These  challenges  would  be  even  more  severe  in  the  context  of  a 
highly  complex  major  defense  acquisition  program  (Fox  &  Miller,  2006).  Further, 
unless  the  development  program  has  been  designed  and  implemented  with 
evaluation  in  mind — with  concomitant  resource  investments — metrics  and  data  are 
typically  unavailable  for  analysis  (Chmielewski  &  Phillips,  2002). 

Such  challenges  are  addressed  generally  but  extensively  in  the  literature  of 
public  program  and  policy  evaluation  (see,  for  example,  Babbie,  2009;  Langbein, 
2006;  Mohr,  1995;  Poister,  2003). 

The  GAO  (2010b)  recently  addressed  the  issue  of  measuring  the  effects  of 
training  in  the  defense  acquisition  workforce.  The  GAO  reported  that  DAU,  which 
has  authority  for  acquisition  training  in  the  DoD,  measures  training  effectiveness 
using  Kirkpatrick’s  (1994)  four-level  model,  as  follows: 

■  Level  1 :  Reaction — the  trainees’  reaction  to  and  satisfaction  with  the 
training. 

■  Level  2:  Learning — the  extent  to  which  learning  has  occurred  because 
of  the  training. 

■  Level  3:  Behavior — the  application  of  learning  to  work  from  changes  in 
trainee’s  behavior. 

■  Level  4:  Results — the  impact  of  the  training  on  the  agency’s 
organizational  results  (e.g.,  ROI;  increased  profitability). 

The  GAO  noted  that,  while  DAU  attempts  to  measure  its  training 
effectiveness  for  the  first  three  levels,  it  has  not  used  Level  4  assessments 
extensively  (2010b,  pp.  1 4-1 5).  Among  other  findings,  it  concluded  that  the  DoD 
needs  metrics  for  assessing  “how  certification  training  contributes  to  organizational 
performance  results”  (p.  28).  In  its  comments  on  the  report,  the  DoD  disagreed  with 
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this  conclusion,  stating  its  preference  for  existing  metrics  related  to  workforce 
capacity  and  proficiency,  rather  than  new  metrics  on  results  (GAO,  2010b,  p.  39). 

While  the  HRM  literature  focuses  on  the  organizational  effects  of  HR 
development  efforts,  it  has  implications  for  studies  that  would  examine 
organizational  effects  due  to  managerial  attributes  and  competencies,  for  example, 
the  effects  of  PM  professionalism  on  program  outcomes.  Specifically,  the  same 
challenges  (e.g.,  external  validity,  availability  of  data)  are  present  for  the  latter  as  for 
the  former.  These  issues  are  also  reflected  in  business  theorists’  long-standing 
debates  over  the  extent  to  which  managerial  competence  contributes  to  firm 
profitability  (Roquebert,  Phillips,  &  Westfall,  1996;  Skandalis,  Liargovas,  &  Merika, 
2008). 

E.  Summary  of  Literature  Review 

To  summarize  the  main  points  of  this  section,  while  some  assume  that  PM 
professionalism  and  program  outcomes  are  positively  correlated,  no  study  has 
attempted  to  validate  that  assumption.  Proximate  research  comes  from  HRM 
studies  that  seek  to  find  relationships  between  HR  development  programs  and 
organizational  outcomes;  these  are,  however,  fraught  with  challenges  in  both 
method  and  data  and  have  met  with  inconclusive  results.  The  following  question 
remains:  To  what  extent  may  the  benefits  of  PM  professionalism  be  based  on 
evidence  rather  than,  apparently,  on  faith? 
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IV. 


Research  Hypothesis  and  Models 


The  discussion  to  this  point  provides  background  for  the  following  research 
hypothesis:  PM  professionalism  is  positively  correlated  with  program  outcomes. 

This  study  examines  this  hypothesis  through  the  analysis  of  dependent  variables 
that  correspond  to  program  outcomes  and  which  are  presumed  to  vary  according  to 
changes  in  independent  variables  related  to  PM  professionalism.  This  section 
presents  the  development  of  variables  for  multiple  linear  (ordinary  least  squares 
[OLS])  and  logistic  regression  models,  along  with  hypotheses  for  statistical  analysis. 
Considering  the  methodological  issues  mentioned  in  the  introduction  to  this  report, 
the  goal  is  for  robust  testing  across  a  variety  of  specifications. 

A.  Dependent  Variables  (Outcomes) 

Many  have  noted  the  difficulty  in  defining  “success”  in  acquisition  program 
outcomes,  given  the  variety  of  stakeholders  representing,  for  example,  military, 
industry,  political,  and  other  interests  (see,  for  example,  Gansler,  2011a,  pp.  1 29— 
234).  Providentially,  this  study  can  avoid  this  problem,  because  most  of  the 
criticisms  cited  in  prior  chapters  which  focus  on  PMs’  lack  of  professionalism  have 
emphasized  adverse  outcomes  at  the  level  of  individual  programs,  such  as  cost 
increases,  schedule  slips,  and  performance  shortfalls.  Thus,  in  this  study, 
dependent  variables  are  outcomes  for  individual  acquisition  programs  and  are 
functions  of  several  independent  variables  for  those  programs. 

Percent  unit  cost  variance  (PUCV).  At  program  initiation,  initial  baseline  cost 
estimates  must  be  established  against  which  subsequent  cost  estimates  can  be 
compared.  Estimates  are  prepared  in  several  categories  both  for  total  costs  and  unit 
costs  (i.e.,  total  cost  divided  by  quantity  to  be  procured).  Because  both  total  costs 
and  unit  costs  can  differ  significantly  among  programs  (e.g.,  the  unit  cost  for  a 
missile  program  may  be  dozens  of  thousands  of  dollars,  while  for  a  ship,  perhaps 
hundreds  of  millions  of  dollars),  analysis  of  baseline  cost  variances  in  dollar  terms 
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provides  skewed  results.  Accordingly,  this  study  will  use  PUCV  as  a  dependent 
variable.  For  example,  if  in  Year  1  the  estimated  baseline  unit  cost  for  a  program 
was  $1  million,  and  in  Year  2  the  estimate  rose  to  $1 .1  million,  the  variance  was  a 
positive  (and  unfavorable)  10%. 6 

Occasionally,  unit  cost  variance  appears  excessive,  as  in  the  case  of  program 
termination,  when  all  program  costs  would  be  allocated  to  perhaps  only  a  few 
developmental  platforms.  Accordingly,  this  study  sets  plus  and  minus  100% 
variance  as  upper  and  lower  limits  and  eliminates  any  data  points  outside  these 
limits  as  outliers. 

PUCV  will  be  examined  in  the  following  three  forms: 

■  PUCV  PERIOD — PUCV  reported  in  each  SAR  period 

■  PUCV  DURATION — PUCV  in  the  last  reporting  period  of  a  PM’s  tenure 
duration7 

■  PUCV  NOMINAL — PUCV  as  a  nominal  variable  with  binary  values 
“favorable”  (if  PUCV  PERIOD  or  DURATION  is  unchanged  or 
decreases)  or  “unfavorable”  (if  PUCV  PERIOD  or  DURATION 
increases). 

Program  breach.  A  breach  occurs,  and  must  be  reported,  when  an  estimate 
of  a  cost,  schedule,  or  performance  parameter  is  determined  to  be  significantly  less 


6  Many  circumstances  could  cause  such  a  result,  such  as  (1)  cost  increases  due  to  poor  initial 
estimates;  (2)  cost  increases  due  to  desired  and  beneficial  changes  (e.g.,  increased  weapon  system 
capability);  or  (3)  changes  in  quantity  to  be  procured.  The  extent  to  which  a  PM’s  professionalism 
might  be  determined  to  influence  any  such  circumstances  is  problematical. 

Discussion  on  reasons  for  program  cost  growth,  appropriate  cost  metrics,  and  the  idea  that  cost 
growth  in  some  circumstances  may  be  justifiable  and  worthy,  is  reflected  in  the  DoD’s  responses  to 
the  recent  GAO’s  critical  assessments  of  acquisition  programs.  See,  for  example,  GAO,  2011a,  pp. 
153,  179-180. 

7  Conceptually,  this  would  be  the  period  in  which  the  PM  should  have  the  most  influence  over  the 
program’s  outcomes,  according  to  conventional  wisdom. 
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favorable  than  its  baseline  estimate.  Breaches  fall  into  two  main  categories:  Nunn- 
McCurdy  breaches8  and  acquisition  program  baseline  (APB)  breaches. 

A  Nunn-McCurdy  breach  (GAO,  2011b)  occurs  and  must  be  reported  when  a 
major  increase  (ranging  from  15%  to  50%,  depending  on  the  specific  criteria  used)  is 
determined  to  have  occurred  in  a  program’s  current  estimate  of  unit  cost  from  the 
baseline  estimate.  A  Nunn-McCurdy  breach  is  an  indicator  of  potentially  serious 
problems  with  a  program,  and  reporting  a  breach  triggers  a  variety  of  actions  by 
higher  authorities,  including  required  reports  to  Congress  on  corrective  actions  that 
might  include  program  termination. 

An  APB  breach  occurs  and  must  be  reported  when  the  current  estimate  for 
any  of  a  program’s  key  parameters  is  determined  to  exceed  its  threshold  (or 
minimally  acceptable)  value  as  defined  in  the  APB.9  Because  APB  thresholds  are 
the  minimally  acceptable  values  of  the  most  important  program  parameters,  the 
determination  that  any  might  not  be  achieved  represents  a  serious  issue,  requiring 
the  attention  of  higher  authorities,  along  with  appropriate  corrective  action. 

For  this  study,  the  breach  variable  will  take  on  two  nominal  values:  yes  (if  a 
program  has  experienced  any  type  of  breach)  or  no  (if  not).  Obviously,  this 
dependent  variable  is  very  general  and  accounts  for  almost  any  adverse 
circumstance  of  sufficient  importance  to  be  reported  to  higher  authorities. 


8  Named  for  Senator  Sam  Nunn  and  Rep.  David  McCurdy,  co-sponsors  of  the  provisions  enacted  in 
the  Department  of  Defense  Authorization  Act  for  1 982  (1981). 

9  The  APB  is  established  at  program  initiation  and  represents  its  approved  description  in  terms  of 
key  cost,  schedule,  and  performance  parameters,  expressed  as  both  objective  (desired)  and 
threshold  values.  Examples  of  key  parameters  that  might  be  included  in  an  APB  are  as  follows: 

■  Cost — estimated  total  cost  for  each  appropriation  type  (e.g.,  procurement,  military 
construction);  unit  cost;  ownership  cost 

■  Schedule — projected  dates  for  major  milestones,  major  tests,  and  initial  operational  capability 

■  Performance — projected  system  attributes  such  as  operational  availability,  range,  airspeed, 
accuracy 

(See  chapter  2.1.1  of  Defense  Acquisition  University,  2011). 
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B.  Independent  Variables 


The  earlier  discussion  gave  indications  of  several  variables  that,  according  to 
the  research  hypothesis,  are  presumed  to  contribute  to  program  outcomes. 

PM  tenure.  Longer  tenures  are  presumed  to  contribute  to  favorable 
outcomes.  This  is  presented  in  two  ways  for  each  acquisition  program,  depending 
on  the  model  specification:  first,  the  number  of  months  that  a  PM  has  been  in  the 
position  as  of  a  SAR  period  report  date  (TENURE  PERIOD);  and  second,  the  total 
duration  of  the  PM’s  tenure  in  months  (TENURE  DURATION). 

PM  status — military  or  civilian.  Programs  with  civilian  PMs  are  presumed  to 
have  favorable  outcomes;  those  with  military  PMs,  less  so.  This  is  a  nominal 
variable  with  two  values. 

Program  component.  This  variable  denotes  the  program’s  component, 
whether  Air  Force,  Army,  Navy,  or  DoD  (joint  programs)  and  is  a  nominal  variable 
with  one  of  these  four  values.  This  variable  is  an  admittedly  blunt  surrogate  for  PM 
training,  education,  and  experience  levels.  Such  data  were  not  readily  available  for 
individual  PMs,  nor  were  data  available  on  whether  a  particular  PM  required  a 
waiver  from  the  statutory  training,  education,  and  experience  levels  required  for 
assignment.  However,  the  discussion  in  section  II. G.  noted  that  (1)  only  the  Air 
Force  has  a  dedicated  career  path  for  its  officers;  and  (2)  Air  Force  PMs  have  more 
training  and  experience  than  either  Army  or  Navy  PMs,  and  Army  PMs  have  more 
experience  than  Navy  PMs.  Accordingly,  Air  Force  programs  are  presumed  to  have 
better  program  outcomes  than  Army  and  Navy  programs,  and  Army  programs  are 
presumed  to  have  better  outcomes  than  Navy  programs. 
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Null  Hypotheses.  The  following  null  hypotheses  follow:  There  is  no 
relationship  between  program  outcomes  and 

■  H10:  duration  of  PM  tenure, 

■  H20:  PM  status,  and 

■  H30:  program  component. 

C.  Extraneous  Variables 

The  following  independent  variables  may  influence  the  dependent  variables 
but  are  not  of  interest  to  the  question  of  PM  professionalism: 

System  complexity.  Unit  cost  (PAUC)  is  used  as  a  proxy  for  the  complexity  of 
the  system  to  be  acquired.  Systems  with  higher  unit  costs  (e.g.,  ships,  aircraft)  are 
assumed  to  have  higher  levels  of  complexity.  For  simplicity,  each  program  will  take 
on  the  nominal  value  (Q1  -  Q4)  of  the  quartile  in  which  its  PAUC  falls. 

Program  commodity  (e.g.,  ship,  aircraft,  missile).  Some  commodities  may 
have  differing  levels  of  risk  due  to  factors  such  as  technological  maturity  and  the 
state  of  the  U.S.  industrial  base,  with  concomitant  potential  effects  on  outcomes. 

This  variable  takes  on  a  nominal  value  for  one  of  eight  program  commodities  listed 
in  the  next  chapter. 

Program  phase.  This  variable  has  one  of  two  nominal  values:  development 
or  production.  This  variable  is  associated  with  the  relative  maturity  and  risk  of  the 
program,  which  may  be  reflected  in  its  outcomes. 

The  model  includes  a  lagged  dependent  variable  to  account  for  the  prior 
period’s  outcome. 

D.  Multiple  Linear  Regression  Model 

Based  on  the  variables  identified  previously  the  general  model  for  the 
outcome — PUCV  or  breach — of  a  program  during  time  period  t  is  as  follows: 
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Outcomet  =  f(tenuret,  statust,  component,  PAUC  quartile,  commodity,  phaset, 
outcomet-i) 

(Component,  PAUC  quartile,  and  commodity  are  assumed  unchanged  over  a 
program’s  life.) 

Several  different  specifications  of  this  model  will  be  examined  in  the  following 
chapter. 


NAVAL  POSTGRADUATE  SCHOOL 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 


-44  - 


V. 


Data 


This  section  describes  the  source  of  data  for  this  study,  explains  the  data 
elements  that  correspond  to  the  relevant  variables,  and  presents  a  summary  of  the 
data  that  were  collected. 

A.  Source  of  Data 

Data  were  obtained  from  the  Defense  Acquisition  Management  Information 
Retrieval  (DAMIR)  Purview10  system.  Data  were  taken  from  Purview’s  records  of  all 
major  defense  acquisition  program* 11  (MDAP)  Selected  Acquisition  Reports12  (SARs) 
submitted  between  1997 — the  first  year  for  which  Purview  SAR  data  are  available — and 
2010.  The  data  include  any  acquisition  program  that  was  designated  as  MDAP  and 
which  submitted  a  SAR  during  any  of  these  years.13 


10  DAMIR  Purview  is  an  executive  information  system  operated  by  the  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics  (Acquisition  Resources  &  Analysis);  see 
http://www.acq.osd.mil/damir/. 

11  Defined  as  a  DoD  acquisition  program  that  is  designated  as  such  by  the  Secretary  of  Defense  and 
is  not  a  highly  sensitive  classified  program,  that  is  estimated  to  require  an  eventual  total  expenditure 
for  research,  development,  test,  and  evaluation  of  more  than  $365  million  or  an  eventual  total 
expenditure  for  procurement  of  more  than  $2.19  billion  (both  in  FY  2000  constant  dollars) 
(Department  of  Defense,  2008,  p.  3-1). 

12  The  Secretary  of  Defense  must  submit  a  SAR  to  Congress  for  each  MDAP.  The  SAR  reports  the 
status  of  total  program  cost,  schedule,  and  performance,  as  well  as  unit  cost  breach  information. 
SARs  are  submitted  annually  and,  on  an  exception  basis,  quarterly,  when  estimates  for  some  cost 
and  schedule  parameters  exceed  their  targets.  (See  section  10.9  of  Defense  Acquisition  University, 
2011,) 

13  This  study  did  not  include  the  DoD’s  chemical  demilitarization  programs.  While  these  are 
sometimes  included  in  lists  of  acquisition  programs  and  are  required  to  submit  SARs,  they  differ 
significantly  in  that  their  objective  is  to  destroy  weapons  rather  than  acquire  them. 

The  study  also  excluded  the  National  Missile  Defense  program  due  to  its  extremely  high  unit  cost  (in 
excess  of  $20  billion),  which  is  attributable  both  to  its  very  complex  “system  of  systems”  nature  and  to 
its  production  as  one  single  system  (i.e.,  quantity  of  one).  Additionally,  this  program  submitted  only 
one  SAR  in  1999.  Other  programs  are  also  very  complex,  with  very  low  production  quantities  (e.g., 
Warfighter  Information  Network-Tactical);  however,  they  typically  have  submitted  SARs  in  several 
years.  For  this  reason,  they  are  included  in  the  analysis  of  this  study. 
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B.  Data  Elements 


Purview  provided  the  following  for  each  year  in  which  an  MDAP  submitted  a 

SAR: 


PUCV.  The  reported  percentage  variance  between  the  current  estimate  and 
the  baseline  estimate  of  Program  Acquisition  Unit  Cost  (PAUC)14  is  reported  in  the 
SAR.  Estimates  are  in  base  year  dollars,  where  the  base  year  is  typically  the  year 
in  which  the  MDAP  was  approved  and  its  baseline  established. 

Breach.  Details  on  any  type  of  breach  that  occurred  during  the  report  period 
are  given  in  the  SAR. 

Tenure  and  Status.  The  SAR  provides  identifying  information  (assignment 
date  and  rank,  if  military)  for  the  current  PM.  PM  tenure  for  the  current  reporting 
period  (TENURE  PERIOD)  is  the  difference  in  months  between  the  PM’s 
assignment  date  and  the  date  of  the  SAR.  PM  total  tenure  (TENURE  DURATION)  is 
taken  as  the  difference  between  the  PM’s  assignment  date  and  that  of  his  or  her 
successor. 

Component.  The  SAR  identifies  each  program’s  component. 

Phase.  The  SAR  gives  information  on  program  milestones,  which  allows  for  a 
determination  of  whether  a  program  is  in  the  development  or  production  phase. 

PAUC.  PAUC  is  reported  in  each  SAR  in  both  base  year  and  then  year 
dollar.  This  study  uses  base  year  dollars. 


14  Defined  as  the  estimated  cost  of  the  development,  procurement,  and  military  construction 
necessary  to  acquire  a  system,  divided  by  the  total  number  of  fully  configured  end  items  (to  include 
research  and  development  units)  to  be  bought  through  the  life  of  the  program. 
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Commodity.  The  SAR  classifies  each  program’s  commodity  according  to  the 
type  of  weapon  or  system  it  acquires.  For  this  study,  the  following  eight 
commodities  were  used: 

■  aircraft  (fixed-  and  rotary- winged,  including  unmanned  aerial  vehicles); 

■  ships  (surface  craft  and  submarines);  ground  vehicles  (tracked  and 
wheeled); 

■  missiles  (air  and  surface); 

■  munitions  (bombs,  artillery  projectiles,  warheads); 

■  command,  control,  communications  and  intelligence  (C3I;  radios, 
information  systems); 

■  space  (satellites,  launch  vehicles);  and 

■  other  (systems  such  as  those  for  soldier  support  which  do  not  fit  in  the 
other  commodities). 

C.  Data  Summary 

This  study  used  Minitab  16  for  all  data  analysis. 

Overview.  Table  1  lists  numbers  of  SAR  submissions  and  programs  by 
commodity  and  component  for  the  years  1997-2010. 

Table  1.  Number  of  SAR  Submissions  and  Programs  by  Commodity  and 

Component 


SAR 

submissions 

Air 

Force 

Army 

DoD 

Navy 

Programs 

Air 

Force 

Army 

DoD 

Navy 

Space 

128 

0 

0 

15 

Space 

16 

0 

0 

2 

Aircraft 

145 

81 

9 

145 

Aircraft 

19 

13 

1 

17 

Missile 

35 

66 

8 

71 

Missile 

3 

8 

4 

8 

C3I 

26 

93 

42 

52 

C3I 

7 

27 

5 

12 

Ship 

0 

0 

0 

128 

Ship 

0 

0 

0 

18 

Munitions 

39 

38 

0 

31 

Munitions 

6 

6 

0 

3 

Ground 

Vehicle 

0 

59 

0 

13 

Ground 

Vehicle 

0 

7 

0 

2 

Other 

15 

5 

6 

2 

Other 

1 

1 

3 

1 
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Figure  1  shows  numbers  of  SAR  submissions  for  programs  by  phase  and 
component. 


Figure  1.  SAR  Submissions  by  Phase  (D  =  Development;  P  =  Production) 

and  Component 

Program  Complexity  (PAUC  Quartile).  Figure  2  provides  bar  charts  depicting 
percentages  of  SAR  submissions  per  quartile  according  to  component  and 
commodity.  As  expected,  Navy  and  Air  Force  programs,  representing  a 
preponderance  of  ship,  space,  and  aircraft  programs,  appear  to  have  the  highest 
levels  of  complexity. 


Figure  2.  PAUC  Quartile  by  Component  and  Commodity 
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VI. 


Data  Analysis 


This  section  presents  descriptive  statistics,  graphical  data  displays,  and 
results  of  regression  analysis,  together  with  commentary  on  how  these  relate  to  the 
study’s  hypotheses.  It  begins  with  data  on  PMs  and  then  moves  to  data  on 
outcomes  (the  study’s  dependent  variables). 

A.  PM  Data 

PM  Population.  Earlier  discussion  documented  the  interest  of  various 
commissions  and  study  groups  in  the  composition  of  the  PMs  according  to  status 
(i.e.,  military  or  civilian).  Table  2  presents  data  from  the  timeframe  of  this  study 
(1997-2010)  for  the  number  and  percent  of  PMs  by  status  and  according  to  their 
component  and  commodity.  Table  3  provides  a  comparison  of  PM  status  between 
this  present  study  and  the  1990  HASC  Report  discussed  earlier. 

The  data  indicate  roughly  a  10%  increase  in  civilian  PMs  of  major  programs 
over  the  past  twenty  years.  Considering  that  over  80%  of  PM  positions  remain 
occupied  by  military  members,  however,  this  increase  likely  would  be  insufficient  to 
satisfy  past  commissions  and  groups  that  advocated  increased  levels  of  civilian 
PMs.  No  striking  differences  are  apparent  from  the  distribution  of  PMs  by 
component  and  commodity.15 


15  The  sole  component/commodity  area  in  Table  2  in  which  civilian  PMs  were  a  majority — Air  Force 
“other”  programs — achieved  this  distinction  by  having  four  of  five  civilian  PMs  in  only  one  program — 
the  National  Airspace  System — during  the  1997-1010  timeframe.  Thus,  the  increase  in  civilians 
appears  to  be  relatively  limited. 
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Table  2.  PM  Data  by  Status,  Component,  and  Commodity 


Aircraft 

Space 

C3I 

Missile 

Munitions 

Ground 

Vehicle 

Ship 

Other 

Total 

Military 

Air  Force 

53 

46 

17 

18 

10 

0 

0 

1 

145 

Army 

33 

0 

32 

20 

19 

23 

0 

2 

129 

DoD 

5 

0 

17 

3 

0 

0 

0 

3 

28 

Navy 

51 

3 

16 

30 

11 

4 

43 

1 

159 

Total 

142 

49 

82 

71 

40 

27 

43 

7 

461 

Civilian 

Air  Force 

14 

11 

2 

3 

8 

0 

0 

4 

42 

Army 

5 

0 

4 

7 

0 

3 

0 

0 

19 

DoD 

0 

0 

2 

1 

0 

0 

0 

1 

4 

Navy 

4 

2 

7 

2 

1 

2 

8 

0 

26 

Total 

23 

13 

15 

13 

9 

5 

8 

5 

91 

Percent 

Military 

Air  Force 

79 

81 

89 

86 

56 

20 

Army 

87 

89 

74 

100 

88 

100 

DoD 

100 

89 

75 

75 

Navy 

93 

60 

70 

94 

92 

67 

84 

100 

Total 

86 

79 

85 

85 

82 

84 

84 

58 

84 

Table  3.  Comparison  of  PM  Status:  Present  Study  and  the  1990  HASC  Report 


Percent  Military 

This  study 

1 990  HASC  Report 

Air  Force 

78 

96 

Army 

87 

96 

DoD 

88 

— 

Navy 

86 

94 

Total 

84 

96 

Figure  3  indicates  that  military  PMs  tend  to  be  assigned  more  frequently  to 
programs  with  the  highest  levels  of  complexity  (i.e. ,  highest  PAUC)  than  civilian 
PMs,  especially  in  the  Army  and  Navy.  A  two-sample  T-test  confirmed  that  PAUC 
for  programs  with  military  PMs  ( M  =  $497  million,  SD  =  $1,477  million)  is  significantly 


NPS 
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higher  than  those  with  civilian  PMs  ( M  =  $224  million,  SD  =  $489  million)  (t( 788)  =  - 
4.59 ,p<  .01). 


Figure  3.  PM  Status  and  PAUC  Quartile 

PM  Tenure.  This  study  has  documented  concerns  with  frequent  PM  turnover 
(short  tenures  in  position)  as  contributing  to  unfavorable  program  outcomes. 
Statistics  on  PM  tenure  (TENURE  PERIOD  and  TENURE  DURATION)  are  given  in 
Table  4. 


Table  4.  Summary  Statistics  for  PM  Tenure 


Variable 

Mean 

Std 

Dev 

Median 

TENURE  PERIOD  (tenure  for  the  SAR  period  in 
months)  (N  =  1251) 

17.3 

12.92 

16.0 

TENURE  DURATION  (total  tenure  duration  in 
months)  (N  =  399) 

32.4 

14.17 

35.0 
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Figures  4  through  7  provide  graphical  information  in  the  forms  of  histograms 
and  “box  and  whisker”  plots16  of  PM  TENURE  DURATION  according  to  component, 
commodity,  phase,  and  status. 
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Figure  4.  PM  TENURE  DURATION  and  Component 


16  Minitab  uses  the  quartile  method  for  calculating  box  endpoints.  The  top  of  the  box  is  the  third 
quartile  (Q3);  75%  of  the  data  values  are  less  than  or  equal  to  this  value.  The  bottom  of  the  box  is  the 
first  quartile  (Q1);  25%  of  the  data  values  are  less  than  or  equal  to  this  value.  The  upper  “whisker” 
extends  to  the  highest  data  value  within  the  upper  limit  (upper  limit  =  Q3  +  1 .5  (Q3  -  Q1 )),  and  the 
lower  whisker  extends  to  the  lowest  value  within  the  lower  limit  (lower  limit  =  Q1-  1.5  (Q3  -  Q1)). 
Values  beyond  the  whiskers  are  outliers. 
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Figure  6.  PM  TENURE  DURATION  and  Status  (M  =  military;  C  =  civilian) 
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Figure  7.  PM  TENURE  DURATION  and  Phase  (Development  or  Production) 

Table  5  provides  a  comparison  of  PM  tenure  durations  between  this  present 
study  and  the  1990  HASC  Report  discussed  earlier.  On  average,  PM  tenure 
durations  have  remained  relatively  unchanged  over  the  past  twenty  years.  Average 
tenure  has  increased  in  the  Air  Force  and  Army  and,  while  decreasing  in  the  Navy, 
average  tenure  remains  the  longest  in  that  component. 
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Table  5.  Comparison  of  PM  Tenure:  Present  Study  and  the  1990  HASC  Report 


PM  Average  Tenure  (months) 

This  study 

1990  HASC  Report 

Air  Force 

28.5 

25 

Army 

33.2 

29 

DoD 

28.8 

-- 

Navy 

36.0 

41 

Total 

32.4 

32 

Discussion  of  PM  Tenure.  No  significant  differences  are  apparent  regarding 
commodity  or  phase. 

Figure  6  bears  comment,  as  it  indicates  that  tenure  durations  tend  to  be 
longer  for  military  PMs.  This  runs  counter  to  the  idea  that  civilians  should  be 
preferred  for  PM  positions,  since  they  would  (supposedly)  have  less  frequent 
reassignments  and  longer  tenure  durations  than  military  members.  One  possible 
reason  for  this  phenomenon  is  that  civilians  might  be  appointed  as  “interim”  PMs, 
with  corresponding  short  tenures,  pending  assignments  of  military  members.  If 
valid,  this  explanation  would  put  the  recent  increase  in  civilian  PMs  in  a  different 
light. 

B.  Dependent  Variables — Graphs 

This  section  provides  graphical  analyses  of  each  the  study’s  dependent 
variables  in  relation  to  the  independent  and  extraneous  variables. 

PUCV.  PUCV  PERIOD  and  PUCV  DURATION  are  examined  together 
because  of  their  similarity.  Table  6  provides  summary  statistics. 
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Table  6.  Summary  Statistics  for  Percent  Unit  Cost  Variance  (PUCV) 


Variable 

N 

Mean 

Std  Dev 

Median 

PUCV  PERIOD  (N  =  1168) 

1169 

-0.39 

15.78 

0.0 

PUCV  DURATION  (N  =  404) 

405 

0.23 

14.42 

0.02 

Figure  8  shows  scatter  plots  for  PUCV  PERIOD  and  DURATION  according  to 
reporting  periods  in  the  timeframe  of  interest  (1997-2010).  These  outcomes  appear 
to  be  uncorrelated  with  time,  a  phenomenon  which,  while  not  formally  addressed  in 
this  study,  has  possible  relevance  to  the  professionalization  issue.  Specifically,  if  the 
DoD  has  invested  substantial  resources  in  PM  professionalization  efforts  over  the 
past  twenty  years,  one  might  expect  to  see  improved  outcomes.  This  appears, 
however,  not  to  be  the  case. 


Figure  8.  PUCV  and  Time 

Figure  9  shows  scatter  plots  for  these  two  in  relation  to  PM  TENURE 
PERIOD  and  TENURE  DURATION,  respectively.  Little  correlation  is  evident. 


VRAtV^T'A  PtR  SClfMUui 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-55- 


Figure  9.  PUCV  as  a  Function  of  PM  Tenure 

Figures  10  through  14  show  box  plots  for  PUCV  PERIOD  and  PUCV 
DURATION  in  relation  to  component,  status,  commodity,  phase,  and  PAUC  Quartile. 
Again,  none  of  these  exhibits  evident  correlation. 


Figure  10.  PUCV  as  a  Function  of  Component 
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Figure  11.  PUCV  as  a  Function  of  Status  (C  =  civilian;  M  =  Military) 


Figure  12.  PUCV  as  a  Function  of  Commodity 


PHQSE 


PHQSE 


Figure  13.  PUCV  as  a  Function  of  Phase  (D  =  development;  P  =  production) 
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Figure  14.  PUCV  as  a  Function  of  PAUC  Quartile 

Discussion  of  PUCV.  Regarding  Figure  10,  a  one-way  ANOVA  was  used  to 
test  for  PUCV  PERIOD  differences  among  the  four  components.  PUCV  PERIOD 
differed  significantly  across  components  (F( 3,  1164)  =  4.01,  p  <  .01).  Tukey’s 
comparisons  (p  =  .05)  indicate  that  the  Army  (M  =  -2.87,  SD  =  20.5 )  had  significantly 
lower  (favorable)  PUCV  PERIOD  than  both  the  Air  Force  ( M  =  0.29,  SD  =  15.85)  and 
the  Navy  ( M  =  0.95,  SD  =  11 .22),  but  the  Army’s  PUCV  PERIOD  was  not 
significantly  different  than  the  DoD’s  ( M  =  -1 .29;  SD  =  1 3.54).  Comparisons  of 
PUCV  PERIOD  for  the  DoD,  Air  Force,  and  Navy  were  not  statistically  significant. 

This  finding  disconfirms  H3o  regarding  a  correlation  between  program 
outcomes  and  component,  but  not  in  the  manner  predicted.  The  research 
hypothesis  predicted  that  the  Air  Force  would  have  the  highest  levels  of  favorable 
outcomes,  but  the  data  indicate  the  highest  levels  in  the  Army. 

Figure  1 1  bears  comment,  since  it  appears  that  the  variance  of  PUCV  is 
higher  for  military  PMs  than  for  civilian  PMs.  An  F-test  for  equal  variances  of  PUCV 
PERIOD  confirms  that  the  variance  for  military  PMs  (264.2)  is  significantly  higher 
than  that  for  civilians  (155.9;  F  =  0.59,  999,  167,  p  <  .01 ).  Additionally,  a  two-sample 
T-test  reveals  that  PUCV  PERIOD  for  civilian  PMs  ( M  =  -2.4,  SD  =  12.5)  is 
significantly  lower  than  for  military  PMs  (M  =  -0. 1 ,  SD  =  16.3;  t( 27 1 )  =  -2. 1 2,  p  <  .05). 
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This  finding  disconfirms  H20  regarding  a  correlation  between  program 
outcomes  and  status.  As  predicted  by  the  research  hypothesis,  PUCV  outcomes  for 
civilian  PMs  are  significantly  more  favorable  than  for  military  PMs,  and  they  have 
significantly  less  variance. 

This  finding  is  tempered,  however,  by  evidence  presented  earlier  that  civilians 
tend  to  be  assigned  to  programs  with  lower  complexity,  and  by  evidence  that  PUCV 
is  more  favorable  in  less  complex  programs.  A  one-way  ANOVA  was  used  to  test 
for  PUCV  PERIOD  differences  among  the  four  PAUC  quartiles.  PUCV  PERIOD 
differed  significantly  across  quartiles  (F(3,  1164)=  17.14,  p<  .01).  Tukey’s 
comparisons  (p  =  .05)  indicate  that  Q1  (M  =  -4.48,  SD  =  19.35)  and  Q2  (M  =  -2.99, 
SD  =  16.44)  had  significantly  lower  (favorable)  PUCV  PERIOD  than  both  Q3  ( M  = 
2.98,  SD  =  1 2.26)  and  Q4  (M  =  2.21 ,  SD  =  1 3.20).  PUCV  PERIOD  differences 
between  Q1  and  Q2  were  not  significant,  nor  between  Q3  and  Q4.  Further 
investigation  is  warranted. 

Since,  as  noted  earlier,  tenure  lengths  for  civilian  PMs  are  in  general  shorter 
than  those  for  military  PMs,  tenure  duration  may  not  be  a  significant  factor;  this  will 
be  examined  subsequently  via  OLS  regression. 

PUCV  NOMINAL.  Instances  of  favorable  and  unfavorable  PUCV  are  shown 
in  Table  7. 


Table  7.  Counts  of  PUCV  NOMINAL 


Variable 

N  FAVORABLE 

N  UNFAVORABLE 

PUCV  NOMINAL  PERIOD 

614 

554 

PUCV  NOMINAL  DURATION 

202 

203 

Figure  15  shows  the  percentage  distribution  of  PUCV  NOMINAL  PERIOD 
according  to  PAUC  quartile.  As  might  be  expected,  programs  with  higher  PAUC 
(complexity)  tend  to  have  higher  levels  of  unfavorable  outcomes. 
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Figure  15.  PUCV  NOMINAL  PERIOD  and  PAUC  Quartile 

Figures  16-18  depict  the  distribution  of  PUCV  NOMINAL  according  to 
variables  of  interest.  Little  is  remarkable  about  these  displays,  except  perhaps  that 
for  aircraft  programs,  unfavorable  PUCV  NOMINAL  counts  exceed  favorable  counts 
(Figure  18). 


Figure  16.  PUCV  NOMINAL  PERIOD  and  DURATION  (Favorable  and 

Unfavorable),  Component,  and  Status  (Military  =  M;  Civilian  =  C) 
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Figure  17.  PUCV  NOMINAL  PERIOD  and  DURATION  (Favorable  and 
Unfavorable)  and  Tenure  (Period  and  Duration) 


Figure  18.  PUCV  NOMINAL  PERIOD  and  DURATION  (Favorable  and 

Unfavorable)  and  Commodity 

Discussion  of  PUCV  Nominal.  Regarding  Figure  16,  a  two-proportions  test 
determined  a  significant  difference  in  the  proportions  of  favorable  instances  of  PUCV 
NOMINAL  between  the  civilian  and  military  PM  samples  (Z  =  3.91,  p  <  .01).  This 
disconfirms  H2o  and  supports  the  research  hypothesis  that  program  outcomes  are 
positively  correlated  with  civilian  PMs. 
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Breaches.  Figure  19  depicts  the  distribution  of  numbers  of  SAR  breaches 
during  reporting  periods  from  1997-2010. 17  Again,  as  mentioned  in  the  discussion 
on  PUCV,  one  might  have  expected  to  see  a  downward  trend,  considering  recent 
emphasis  on  PM  professionalization;  this  appears,  however,  not  to  be  the  case. 


Figure  19.  Counts  of  SAR  Breaches,  1997-2010 

Figures  20-23  depict  the  distribution  of  breaches  during  1997-2010 
according  to  variables  of  interest.  These  are  generally  unremarkable  except  for  the 
case  that  breaches  exceed  non-breaches  for  DoD  programs  and  for  space  programs 
(Figure  22). 


17  The  bars  with  the  highest  counts  correspond  to  annual  SAR  submissions  in  December.  The  bars 
with  lower  counts  represent  exception  SARs,  as  described  earlier.  Annual  SARs  were  not  submitted 
in  either  2000  or  2008. 
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Figure  20.  Counts  of  Breaches  (Yes  or  No)  and  PM  Status 

(Military  =  M  or  Civilian  =  C) 


Figure  21.  Breaches  (Yes  or  No)  and  PM  Tenure  (Period  and  Duration) 
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Figure  22.  Breaches  by  Component  and  Commodity 
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Figure  23.  Breaches  by  Phase  (D  =  Development;  P  =  Production) 

and  PAUC  Quartile 

Discussion  of  breaches.  Regarding  Figure  20,  a  two-proportions  test 
determined  a  significant  difference  in  the  proportions  of  reported  breaches  between 
the  civilian  and  military  PM  samples  (Z  =  -2.77,  p  <  .01).  This  disconfirms  H20  and 
supports  the  research  hypothesis  that  program  outcomes  are  positively  correlated 
with  civilian  PMs.  When,  however,  the  same  tests  were  conducted  within  each 
quartile,  a  significant  difference  (Z  =  -2.12,  p  <  .01 )  was  found  only  in  quartile  2  (Q2); 
no  significant  difference  was  found  in  the  proportion  of  breaches  according  to  status 
in  Q1 ,  Q3,  or  Q4. 

Summary — Graphical  analysis  of  dependent  variables.  This  section  has 
given  an  extensive  set  of  depictions  of  relationships  between  the  dependent  and 
independent  variables  of  interest.  For  the  most  part,  few  correlations  are  evident 
from  historical  SAR  data. 

The  notable  exception  that  emerges  is  the  relationship  between  program 
outcomes  and  status:  programs  with  a  civilian  PM  are  associated  with  favorable 
outcomes  to  a  greater  extent  than  those  with  a  military  PM.  Because  tenure 
durations  for  military  PMs  tend  to  exceed  those  of  civilian  PMs,  this  difference  is 
likely  due  to  factors  other  than  tenure.  The  data  indicate  that  military  PM’s  manage 
programs  with  higher  complexity  than  civilians,  though  the  effect  of  program 
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complexity  on  outcomes  is  mixed.  Further  attention  is  given  to  this  issue  in  the 
following  section. 


C.  Regression  Analysis 

This  section  presents  results  of  OLS  regressions  for  the  dependent  variables 
representing  program  outcomes  and  the  independent  variables  of  interest.  Recall 
that  the  general  model  for  the  outcome — PUCV  or  breach — of  a  program  during  time 
period  t  is  as  follows: 


Outcomet  =  f(tenuret,  statust,  component,  PAUC  quartile,  commodity,  phaset, 

outcomet-i) 

where  component,  PAUC  quartile,  and  commodity  are  assumed  unchanged  over  a 
program’s  life.  (As  nominal  variables,  status,  component,  commodity,  and  phase  are 
made  indicator  variables  in  the  regression  equations.)  For  each  outcome,  the  model 
for  program  i  and  time  t  is  given  by  the  following: 

PUCV  PERIODit  =  ait  +  PiTENURE  PERIODit  +  p2STATUSit  + 

PsCOMPONENT  +  p4PAUC  OUARTILE  +  p5COMMODITY  + 

PePHASEit  +  PUCV  PERIODit- 1 

PUCV  DURATIONS  =  ait  +  pITENURE  DURATIONS  +  p2STATUSit  + 
P3COMPONENT  +  P4PAUC  OUARTILE  +  p5COMMODITY  + 
p6PHASEit  +  PUCV  DURATIONit-1 

PUCV  NOMINAL  PERIODit  =  ait+  PiTENURE  PERIODit  +  p2STATUSit  + 
PsCOMPONENT  +  p4PAUC  OUARTILE  +  p5COMMODITY  + 
PePHASEit  +  PUCV  NOMINAL  PERIODit- 1 

PUCV  NOMINAL  DURATIONS  =  ait  +  piTENURE  DURATIONS  + 
P2STATUSit  +  P3COMPONENT  +  p4PAUC  OUARTILE  + 
P5COMMODITY  +  p6PHASEit  +  PUCV  NOMINAL  DURATIONit-1 

BREACHit  =  ait  +  piTENURE  PERIODit  +  p2STATUSit  +  p3COMPONENT  + 
P4PAUC  OUARTILE  +  p5COMMODITY  +  p6PHASEit  +  BREACHit- 1 

Linear  Regression  Results.  Tables  8  and  9  provide  unstandardized  factor 
coefficients  from  progressive  OLS  regressions  for  PUCV  PERIOD  and  PUCV 
DURATION,  respectively. 
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Table  8.  Regression  Results  for  Percent  Unit  Cost  Variance  Period  (PUCV 

PERIOD) 


Predictor 

Coefficient 

(Standard  Error) 

Constant 

1.065 

(0.767) 

-0.929 

(1.356) 

-0.307 

(1.434) 

0.261 

(1.307) 

0.225 

(1.686) 

-0.151 

(1.693) 

-3.98* 

(2.117) 

TENURE  PERIOD 

-0.085** 

(0.036) 

-0.086** 

(0.036) 

-0.088** 

(0.036) 

-0.040 

(0.033) 

-0.044 

(0.033) 

-0.045 

(0.033) 

-0.044 

(0.033) 

Status:  Military 
(Relative  to  Civilian) 

2.338* 

(1.312) 

2.577* 

(1.324) 

1.250 

(1.215) 

1.086 

(1.238) 

1.025 

(1.237) 

0.757 

(1.233) 

Component  (Relative 
to  Air  Force) 

Army 

-3.351*** 

(1.220) 

-1.589 

(1.128) 

-1.243 

(1.331) 

-1.499 

(1.335) 

-0.637 

(1.342) 

DoD 

-1.946 

(2.391) 

0.215 

(2.218) 

1.599 

(2.443) 

0.995 

(2.457) 

3.248 

(2.538) 

Navy 

0.592 

(1.132) 

0.395 

(1.036) 

-0.076 

(1.212) 

-0.285 

(1.215) 

-0.866 

(1.241) 

PUCV  PERIOD  Lag 

0.538*** 

(0.028) 

0.529*** 

0.029 

0.526*** 

(0.029) 

0.508*** 

(0.029) 

Commodity  (Relative 
to  Space) 

Aircraft 

0.827 

(1.584) 

0.517 

(1.589) 

-0.622 

(1.761) 

C3I 

-1.818 

(1.983) 

-2.309 

(1.994) 

-1.888 

(2.066) 

Ground  Vehicle 

0.514 

(2.393) 

0.475 

(2.390) 

1.665 

(2.509) 

Missile 

0.037 

(1.838) 

-0.173 

(1.838) 

1.963 

(2.017) 

Munitions 

0.089 

(1.990) 

-0.144 

(1.990) 

3.752 

(2.335) 

Ship 

2.054 

(2.093) 

2.413 

(2.097) 

2.140 

(2.182) 

Other 

-0.536 

(3.361) 

-1.407 

(3.383) 

-1.311 

(3.466) 

Phase:  Development 
(Relative  to 

Production) 

1.839** 

(0.898) 

1.857** 

(0.900) 

PAUC  Quartile 
(Relative  to  Q1) 

Q2 

1.978 

(1.481) 

Q3 

6.625*** 
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(1.601) 

Q4 

4.930*** 

(1.747) 

R-squared 

.005 

.008 

.019 

.281 

.285 

.288 

.302 

Adj.  R-squared 

.004 

.006 

.014 

.277 

.275 

.278 

.290 

1.  *,  **,  ***  indicate  significance  at  .90,  .95,  and  .99  levels,  respectively. 

2.  Number  of  observations:  998  (when  the  lag  factor  is  included);  1168  (when  the  lag  factor  is  omitted). 

3.  Variance  inflation  factors  ranged  from  a  maximum  of  3.89  to  a  minimum  of  1.03  ( M  =  2.16,  SD  =  .89), 
indicating  that  multicollinearity  was  not  high. 

4.  The  Durbin  Watson  statistic  value  (2.15)  indicates  the  absence  of  autocorrelation  in  the  sample. 

Table  9.  Regression  Results  for  Percent  Unit  Cost  Variance  Duration  (PUCV 

DURATION) 


Predictor 

Coefficient 
(Standard  Error) 

Constant 

2.436 

(1.816) 

-0.123 

(2.334) 

0.677 

(2.404) 

-1.224 

(1.733) 

-2.716 

(2.309) 

-2.733 

(2.321) 

-4.001 

(2.755) 

TENURE 

DURATION 

-0.067 

(0.051) 

-0.077 

(0.051) 

-0.073 

(0.053) 

-0.034 

(0.040) 

-0.033 

(0.040) 

-0.033 

(0.040) 

-0.009 

(0.040) 

Status:  Military 
(Relative  to  Civilian) 

3.449* 

(1.983) 

3.573* 

(1.992) 

3.162** 

(1.417) 

3.304** 

(1.443) 

3.304** 

(1.445) 

1.818 

(1.404) 

Component 
(Relative  to  Air 

Force) 

Army 

-3.624* 

(1.903) 

-0.637 

(1.409) 

-2.055 

(1.630) 

-2.068 

(1.639) 

-1.707 

(1.619) 

DoD 

-0.351 

(3.327) 

1.638 

(2.426) 

0.130 

(2.752) 

0.094 

(2.785) 

1.639 

(2.994) 

Navy 

-0.225 

(1.791) 

0.907 

(1.291) 

0.073 

(1.504) 

0.061 

(1.512) 

-0.381 

(1.497) 

PUCV  DURATION 

Lag 

0.753*** 

(0.037) 

0.754*** 

(0.038) 

0.754*** 

(0.038) 

0.839*** 

(0.039) 

Commodity 
(Relative  to  Space) 

Aircraft 

1.739 

(1.998) 

1.715 

(2.020) 

-0.675 

(2.181) 

C3I 

3.245 

(2.491) 

3.202 

(2.543) 

2.962 

(2.674) 

Ground  Vehicle 

5.689* 

(2.994) 

5.682* 

(3.000) 

5.710* 

(3.086) 

Missile 

1.465 

(2.262) 

1.445 

(2.276) 

2.076 

(2.458) 

Munitions 

1.497 

(2.469) 

1.483 

(2.477) 

2.985 

(2.805) 

Ship 

2.729 

2.748 

-0.047 
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(2.671) 

(2.683) 

(2.739) 

Other 

2.752 

(4.093) 

2.715 

(4.121) 

0.704 

(4.106) 

Phase: 

Development 
(Relative  to 
Production) 

0.103 

(1.160) 

0.571 

(1.129) 

PAUC  Quartile 
(Relative  to  Q1) 

Q2 

1.244 

(1.804) 

Q3 

4.303** 

(1.927) 

Q4 

4.801** 

(2.164) 

R-squared 

.004 

.012 

.023 

.557 

.563 

.563 

.627 

Adj.  R-squared 

.002 

.007 

.011 

.549 

.546 

.544 

.607 

1.  *,  **,  ***  indicate  significance  at  .90,  .95,  and  .99  levels,  respectively. 

2.  Number  of  observations:  348  (when  the  lag  factor  is  included);  400  (when  the  lag  factor  is  omitted). 

3.  Variance  inflation  factors  ranged  from  a  maximum  of  4.05  to  a  minimum  of  1 .07  ( M  =  2.20,  SD  =  .90), 
indicating  that  multicollinearity  was  not  high. 

4.  The  Durbin  Watson  statistic  value  (1.93)  indicates  the  absence  of  autocorrelation  in  the  sample. 

Discussion  of  Results.  Several  points  are  evident  from  Tables  8  and  9.  First, 
R-squared  (the  coefficient  of  determination)  is  twice  as  large  for  PUCV  DURATION 
as  for  PUCV  PERIOD,  which  is  likely  attributable  to  the  decrease  in  variation  when 
reporting  PUCV  only  once  at  the  end  of  a  PM’s  tenure  rather  than  reporting  in  each 
year  of  tenure. 

As  expected,  the  lag  variable  in  each  model  and  iteration  is  significantly 
related  to  (p  <  .01 )  and  accounts  for  most  of  the  variance  in  the  dependent  variable. 

The  independent  variables  of  interest — tenure,  status,  and  component — 
explain  only  about  2%  of  the  variance  in  the  outcome,  a  finding  which  is  confirmatory 
of  the  null  hypotheses. 

Despite  this  last  finding,  the  signs  of  the  coefficients  for  two  of  the 
independent  variables  of  interest — tenure  and  status — are  in  all  cases  in  the 
direction  predicted  by  the  research  hypotheses.  That  is,  the  negative  coefficients  for 
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duration  indicate  that  PUCV  decreases  as  tenure  increases,  and  the  positive 
coefficient  for  status  indicates  that  military  PMs  are  associated  with  higher  PUCV 
and  civilian  PMs  with  lower  PUCV.  In  only  some  cases,  however,  are  these 
relationships  significant;  they  are  not  significant  in  either  full  model. 

The  directions  of  coefficient  signs  of  the  other  independent  variable  of 
interest — component — are  not  as  predicted  by  the  research  hypotheses.  Programs 
in  the  Air  Force,  due  to  its  structural  factors  that  promote  PM  professionalization, 
were  predicted  to  have  better  outcomes  than  the  other  three  components.  The 
negative  coefficients  for  the  Army  in  all  cases,  however,  indicate  that  PUCV  tends  to 
be  lower  in  Army  programs;  the  same  can  be  said  in  some  cases  for  Navy  and  DoD 
programs. 

Regarding  the  extraneous  independent  variables,  a  few  notable  points  can  be 
made.  First,  the  signs  of  coefficients  for  phase  and  PAUC  quartile  are  in  the 
expected  direction:  PUCV  in  the  production  phase  is  expected  to  be  lower  than  in 
development,  and  PUCV  is  expected  to  increase  as  PAUC  increases.  Second,  in 
both  models,  when  a  program  is  in  the  top  two  quartiles  (Q3  and  Q4),  that 
relationship  to  PUCV  is  significant;  thus,  higher  complexity  is  significantly  related  to 
unfavorable  outcomes.  Third,  phase  is  significantly  related  to  outcomes  in  one  of 
the  models  (for  PUCV  PERIOD);  thus,  developmental  programs  are  associated  with 
unfavorable  outcomes.  Finally,  little  can  be  said  regarding  commodity,  other  than 
the  obvious  conclusion  that  it  has  little  relationship  to  or  explanatory  power  for  the 
dependent  variable. 

The  results  are  statistically  secure  only  for  a  few  of  the  factors,  in  the  sense 
that  most  of  the  standard  errors  exceed  the  magnitude  of  their  coefficients. 

To  summarize,  these  models  must  be  judged  to  have  very  little  explanatory 
power  for  the  outcomes  PUCV  PERIOD  and  PUCV  DURATION. 
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Logistic  Regression  Results.  Tables  10  through  18  give  results  of  binary 
logistic  regressions  for  PUCV  NOMINAL  PERIOD  (favorable  or  unfavorable),  PUCV 
NOMINAL  DURATION  (favorable  or  unfavorable),  and  BREACH  (yes  or  no).  For 
each  of  these  three  dependent  variables,  a  table  of  coefficients,  a  table  of  observed 
and  expected  frequencies,  and  a  table  of  measures  of  association  are  shown. 
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Table  10.  Logistic  Regression  Results  for  PUCV  NOMINAL  PERIOD:  Coefficients 


Predictor 

Coefficient 

(Standard 

Error) 

Odds 

Ratio 

Constant 

-1.191*** 

(0.363) 

TENURE  DURATION 

0.001 

(0.001) 

1.00 

Status:  Military  (Relative  to  Civilian) 

0.423** 

(0.214) 

1.53 

Component  (Relative  to  Air  Force) 

Army 

0.131 

(0.228) 

1.14 

DoD 

0.032 

(0.436) 

1.03 

Navy 

0.012 

(0.210) 

1.01 

PUCV  NOMINAL  PERIOD  Lag 

1.748*** 

(0.143) 

5.74 

Commodity  (Relative  to  Space) 

Aircraft 

-0.394 

(0.299) 

0.67 

C3I 

-0.807** 

(0.352) 

0.45 

Ground  Vehicle 

-0.445 

(0.422) 

0.64 

Missile 

0.102 

(0.343) 

1.11 

Munitions 

-0.638 

(0.399) 

0.53 

Ship 

-0.542 

(0.366) 

0.58 

Other 

-0.198 

(0.579) 

0.82 

Phase:  Development  (Relative  to 
Production) 

-0.045 

(0.153) 

0.96 

PAUC  Quartile  (Relative  to  Q1) 

Q2 

0.141 

(0.250) 

1.15 

Q3 

0.670** 

(0.270) 

1.95 

Q4 

0.299 

(0.290) 

1.35 

1.  *,  **,  ***  indicate  significance  at  .90,  .95,  and  .99  levels,  respectively. 

2.  Counts:  Unfavorable — 492  (event);  Favorable — 516 

3.  Goodness-of-Fit  Test  (Pearson):  Chi-Square  =  852.14,  df  =  839,  p  = 
0.369 
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Table  11.  Observed  and  Expected  Frequencies:  PUCV  NOMINAL  PERIOD 


Group 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Unfavorable 

Observed 

20 

26 

26 

30 

41 

44 

74 

72 

78 

81 

Expected 

17.5 

23.5 

28.1 

32.1 

28.3 

54.5 

66.7 

72.3 

78.2 

80.9 

Favorable 

Observed 

80 

75 

75 

71 

60 

57 

26 

29 

23 

20 

Expected 

82.5 

77.5 

72.9 

68.9 

62.7 

46.5 

33.3 

28.7 

22.8 

20.1 

Table  12.  Measures  of  Association:  PUCV  NOMINAL  PERIOD  and  Predicted 

Probabilities 


Pairs 

Number 

Percent 

Summary  Measures 

Concordant 

191721 

75.5 

Somers’  D:  0.52 

Discordant 

60632 

23.9 

Goodman-Kruskal  Gamma:  0.52 

Ties 

1519 

0.6 

Kendall’s  Tau-a:  0.26 

Total 

253872 

100.0 
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Table  13.  Logistic  Regression  Results  for  PUCV  NOMINAL  DURATION 

Coefficients 


Predictor 

Coefficient 

(Standard 

Error) 

Odds 

Ratio 

Constant 

-2.124*** 

(0.723) 

TENURE  DURATION 

0.003 

(0.010) 

1.00 

Status:  Military  (Relative  to  Civilian) 

0.122 

(0.357) 

1.13 

Component  (Relative  to  Air  Force) 

Army 

-0.084 

(0.424) 

0.92 

DoD 

0.404 

(0.770) 

1.50 

Navy 

-0.356 

(0.394) 

0.70 

PUCV  DURATION  Lag 

2.556*** 

(0.289) 

12.89 

Commodity  (Relative  to  Space) 

Aircraft 

0.642 

(0.561) 

1.90 

C3I 

0.665 

(0.687) 

1.95 

Ground  Vehicle 

0.951 

(0.777) 

2.59 

Missile 

0.717 

(0.634) 

2.05 

Munitions 

1.025 

(0.725) 

2.79 

Ship 

0.321 

(0.708) 

1.38 

Other 

-0.056 

(1.088) 

0.95 

Phase:  Development  (Relative  to 
Production) 

0.150 

(0.294) 

1.16 

PAUC  Quartile  (Relative  to  Q1) 

Q2 

0.451 

(0.474) 

1.57 

Q3 

0.678 

(0.506) 

1.97 

Q4 

0.871 

(0.563) 

2.39 

1.  *,  **,  ***  indicate  significance  at  .90,  .95,  and  .99  levels, 
respectively. 

2.  Counts:  Unfavorable — 183  (event);  Favorable — 151 

3.  Goodness-of-Fit  Test  (Pearson):  Chi-Square  =  312.450,  df  =  297, 
p  =  0.258 
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Table  14.  Observed  and  Expected  Frequencies:  PUCV  NOMINAL  DURATION 


Group 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Unfavorable 

Observed 

8 

9 

9 

12 

10 

21 

25 

30 

27 

32 

Expected 

6.6 

8.7 

9.4 

10.4 

12.5 

21.0 

26.9 

28.7 

28.5 

30.3 

Favorable 

Observed 

25 

25 

24 

21 

24 

12 

8 

4 

6 

2 

Expected 

26.4 

25.3 

23.6 

22.6 

21.5 

12.0 

6.1 

5.3 

4.5 

3.7 

Table  15.  Measures  of  Association:  PUCV  NOMINAL  DURATION  and  Predicted 

Probabilities 


Pairs 

Number 

Percent 

Summary  Measures 

Concordant 

22137 

80.1 

Somers’  D:  0.61 

Discordant 

5359 

19.4 

Goodman-Kruskal  Gamma:  0.61 

Ties 

137 

0.5 

Kendall’s  Tau-a:  0.30 

Total 

27633 

100.0 
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Table  16.  Logistic  Regression  Results  for  BREACH:  Coefficients 


Predictor 

Coefficient 

(Standard 

Error) 

Odds 

Ratio 

Constant 

-0.548 

(0.383) 

TENURE  PERIOD 

0.002 

(0.006) 

1.00 

Status:  Military  (Relative  to  Civilian) 

0.450** 

(0.219) 

1.57 

Component  (Relative  to  Air  Force) 

Army 

-0.084 

(0.233) 

0.92 

DoD 

0.484 

(0.477) 

1.62 

Navy 

-0.178 

(0.215) 

0.84 

BREACH  Lag 

1.951*** 

(0.146) 

7.03 

Commodity  (Relative  to  Space) 

Aircraft 

-0.329 

(0.304) 

.72 

C3I 

-0.575 
(0.  362) 

0.56 

Ground  Vehicle 

-0.226 
(0.  433) 

0.80 

Missile 

-0.827** 

(0.  356) 

0.44 

Munitions 

-1.176*** 

(0.406) 

0.31 

Ship 

-0.048 

(0.374) 

0.95 

Other 

-1 .244* 
(0.692) 

0.29 

Phase:  Development  (Relative  to 
Production) 

0.290* 

(0.155) 

1.34 

PAUC  Quartile  (Relative  to  Q1) 

Q2 

-0.722*** 

(0.260) 

0.49 

Q3 

-0.587** 

(0.279) 

0.56 

Q4 

-0.709** 

(0.306) 

0.49 

1.  *,  **,  ***  indicate  significance  at  .90,  .95,  and  .99  levels, 
respectively. 

2.  Counts:  Yes — 474  (event);  No — 534 

3.  Goodness-of-Fit  Test  (Pearson):  Chi-Square  =  873.171,  df  =  842,  p 
=  0.222 
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Table  17.  Observed  and  Expected  Frequencies:  BREACH 


Group 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Unfavorable 

Observed 

17 

16 

29 

22 

38 

49 

68 

80 

77 

78 

Expected 

15.4 

21.9 

25.2 

27.8 

31.8 

50.7 

68.8 

72.7 

76.4 

83.4 

Favorable 

Observed 

83 

85 

72 

79 

63 

51 

33 

21 

24 

23 

Expected 

84.6 

79.1 

75.8 

73.2 

69.2 

49.3 

32.2 

28.3 

24.6 

17.6 

Table  18.  Measures  of  Association:  BREACH  and  Predicted  Probabilities 


Pairs 

Number 

Percent 

Summary  Measures 

Concordant 

195659 

77.3 

Somers’  D:  0.55 

Discordant 

56409 

22.3 

Goodman-Kruskal  Gamma:  0.55 

Ties 

1048 

0.4 

Kendall’s  Tau-a:  0.27 

Total 

253116 

100.0 

Discussion  of  Logistic  Regression  Results.18  Regarding  the  models’ 
predictive  qualities,  the  goodness-of-fit  tests,  with  p-values  ranging  from  0.222  to 
0.369,  suggest  insufficient  evidence  that  the  models  are  inadequate  to  fit  the  data. 
This  conclusion  is  supported  by  the  general  similarities  between  the  observed  and 
expected  frequencies  in  each  model,  as  well  as  the  relatively  high  ratios  of 
concordant  to  discordant  pairs.  The  summary  measures  of  association  (Somers’  D, 
Goodman-Kruskal  Gamma,  Kendall's  Tau-a)  range  from  0.26  to  0.61 ,  which 


18  Because  these  are  logistic  regressions,  each  of  the  estimated  coefficients  represents  the  log  of  the 
odds  of  an  event,  where  odds  are  defined  as  the  ratio  of  the  event’s  probability  of  occurrence  over  the 
probability  of  nonoccurrence.  (The  “event”  is  an  unfavorable  outcome  [e.g.,  a  PUCV  increase  or 
breach]).  For  example,  in  Table  16,  the  coefficient  of  0.450  for  status  (military)  represents  the  change 
in  the  log  of  P(breach)/P(no  breach)  when  the  PM  is  military,  with  other  variables  held  constant.  The 
estimated  coefficient  of  0.002  for  tenure  period  is  the  change  in  the  log  of  P(breach)/P(no  breach) 
with  a  one-unit  (1  month)  increase  in  duration  period,  with  other  factors  held  constant. 

Regarding  interpretation  of  odds  ratios,  those  close  to  one  indicate  a  minimal  effect  on  the  dependent 
variable.  In  Table  16,  the  coefficient  of  -1.244  and  odds  ratio  of  0.29  for  commodity  “other”  may  be 
interpreted  as  follows:  “Other”  commodity  programs  tend  to  have  fewer  breaches  than  other 
commodities  (i.e. ,  the  negative  sign  of  the  coefficient  indicates  the  event’s  nonoccurrence),  and  the 
odds  of  “other”  programs  having  a  breach  are  29%  of  the  odds  of  the  rest  of  the  commodities  having 
a  breach. 
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indicates  low  to  moderate  predictive  ability.  Overall,  it  cannot  be  said  that  any  of  the 
models  have  either  low  or  high  predictive  qualities. 

Regarding  the  coefficients  and  odds  ratios,  the  results  reflect  generally  the 
same  conclusions  as  the  linear  regressions  above.  Specifically,  the  effect  of 
component  is  mixed,  with  no  evidence  that  outcomes  of  Air  Force  programs  are 
better  than  those  of  other  components.  As  for  the  status  variable,  outcomes  for 
civilian  PMs  are  statistically  better  (p  <  .05)  than  for  military  PMs  in  two  of  the  three 
models.  Results  for  phase  and  PAUC  quartile  are  mixed. 

Finally,  as  in  the  OLS  regressions,  the  factor  for  lag  dominates  other  variables 
in  its  levels  of  significance,  its  odds  ratios,  and  its  contributions  to  the  predictive 
power  of  the  models.  To  illustrate,  when  the  logistic  regression  for  BREACH  (yes  or 
no)  was  run  again  with  all  variables  except  lag,  the  measures  of  association  ranged 
from  only  0.12  to  0.24,  indicating  this  particular  model’s  poor  predictive  quality. 

D.  Data  Analysis — Conclusions 

The  null  hypotheses  may  now  be  revisited  in  light  of  the  graphical  and 
regression  analyses.19  Recall  these  were  framed  as  follows:  There  is  no  relationship 
between  program  outcomes  and 

■  H10:  duration  of  PM  tenure, 

■  H20:  PM  status,  and 

■  H30:  program  component. 

PM  Tenure.  In  only  a  few  regressions  was  PM  tenure  found  to  be 
significantly  related  to  the  outcome.  In  all  regressions,  the  coefficient  was 


19  The  author  conducted  numerous  variations  on  these  models  and  tests,  most  of  which  are  not 
presented  here.  For  example,  one  individual  made  the  suggestion  to  examine  only  the  best 
performing  programs  (or,  alternatively,  the  worst  performers).  These  excursions  failed,  however,  to 
provide  significant  results  and  so  are  omitted. 
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determined  to  be  very  small,  indicating  little  effect  on  the  outcome  for  a  unit  change 
in  duration.  Thus,  H10  should  not  be  rejected. 

PM  Status.  Statistical  tests  indicated  that  outcomes  for  programs  managed 
by  civilian  PMs  were  better  than  those  managed  by  military  PMs.  This  difference 
was  not  related  to  tenure,  and  military  PMs’  programs  had  higher  complexity  than 
those  managed  by  civilians.  Regression  results  were  mixed;  several  but  not  most 
regressions  determined  a  significant  positive  correlation  between  favorable 
outcomes  and  civilian  status.  In  all  regressions,  the  sign  of  the  coefficient  indicated 
better  outcomes  with  civilian  PMs.  Overall,  however,  the  mixed  results  suggest  the 
relationship  is  weak,  and  H20  is  not  rejected. 

Program  Component.  This  hypothesis  was  based  on  the  presumption  that 
outcomes  of  Air  Force  programs  would  be  significantly  better  than  those  of  other 
components.  In  almost  no  cases,  however,  was  this  relationship  evident.  In  fact, 
outcomes  of  Army  programs  tended  to  be  better  than  those  of  the  other  three 
components.  As  noted,  however,  this  is  tempered  by  evidence  that  Army  programs 
tend  to  be  less  complex  than  those  of  the  other  components.  Accordingly,  H30  is  not 
rejected. 

Summary.  On  the  whole,  this  analysis  fails  to  reject  the  null  hypotheses  and 
thus  fails  to  support  the  research  hypothesis  that  PM  professionalization  contributes 
to  improved  program  outcomes.  It  cannot  be  ruled  out,  then,  that  the  conventional 
wisdom  about  PMs  and  their  influence  on  program  outcomes  may  be  entirely  wrong, 
and  that  the  professional  attributes  of  a  PM  may  actually  have  very  little  to  do  with 
the  success  or  failure  of  his  or  her  program. 
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VII. 


Conclusions  and  Recommendations 


This  paper  has  argued  that  conventional  wisdom  portrays  PM  professionalism 
as  a  central  factor  in  the  success  or  failure  of  a  defense  acquisition  program.  It  has 
also  presented  analysis  that  fails  to  support  this  view.  Although  such  inconclusive 
analysis  may  render  conclusions  and  recommendations  problematical,  it  does 
suggest  some  meaningful  points. 

A.  Conclusions 

Further  research  is  needed.  Two  possibilities  remain:  Either  PM 
professionalism  makes  a  significant  difference,  and  the  analysis  here  has  failed  to 
detect  it,  or  it  does  not  make  a  difference.  In  either  case,  more  analysis  with  better 
data  and  models  are  needed.  This  study  found  no  factor,  other  than  a  time  lag 
factor,  that  is  correlated  with  program  outcomes.  This  absence  of  correlation 
between  outcomes  and  any  of  this  study’s  variables,  including  structural  variables 
like  commodity  and  phase,  is  unexpected.  Clearly,  further  research  is  needed  in 
order  to  enable  reasonably  reliable  predictions  of  a  program’s  future  outcomes.20 

A  few  comments  from  the  expert  interviews  indicate  some  alternative  factors. 
Both  Preston  (2011)  and  Soloway  (2011)  suggested  that  the  term  “program 
manager”  is  a  misnomer.  Preston  noted  that  defense  contractors  manage  DoD 
acquisition  programs,  and  DoD  PMs  manage  the  contractors.  Soloway  recalled  a 
conversation  with  a  PM  who  stated,  “We’re  not  doing  program  management 
anymore.  ...  We’re  managing  people,  budget,  and  politics.  ...  I  don’t  even  do  what  I 
was  really  trained  to  do  ...  from  the  technical  side  of  program  management.”  If  valid, 
these  comments  help  explain  why  the  independent  variables  in  this  study  were  poor 
predictors  of  program  outcomes,  since  none  of  the  factors  to  which  Preston  and 


20  For  example,  Brown’s  (2011)  work  in  examining  program  interdependencies  might  contribute  to  a 
better  factor  than  PAUC  to  represent  program  complexity. 
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Soloway  refer  was  included  in  this  analysis.  Accounting  for  these  factors  in  future 
analysis,  however,  would  entail  new  challenges,  for  example,  in  making  them 
operational  in  a  reasonable  model  (e.g.,  how  to  operationalize  an  acquisition 
program’s  political  factors  and  the  PM’s  involvement  in  those?)  and  in  obtaining 
appropriate  data.  In  any  event,  these  interview  comments  suggest  the  need  for  a 
revised  or  expanded  view  of  the  traditional  role  of  PM. 

Other  factors  outweigh  the  PM’s  influence.  Scholars  of  defense  acquisition 
(see,  for  example,  Fox  &  Miller,  2006;  Kronenberg,  1990)  have  noted  the  high  level 
of  complexity  in  defense  acquisition  programs.  Several  interviewees  in  this  study 
(Berteau,  2011;  Etherton,  201 1 ;  Preston,  201 1 ;  Soloway,  2011)  supported  this  view, 
suggesting  that  the  range  and  influence  of  “non-PM”  factors  in  MDAPs  are  large  in 
comparison  to  PM-related  factors.  This  is  consistent  with  DoD’s  recent  “Big  A  -  little 
a”  construct  of  acquisition  consisting  largely  of  activities  that  lie  beyond  the  PM’s 
ability  to  influence,  and  it  helps  explain  why  the  best  managerial  efforts  of  even  the 
most  professional  PMs  would  fail  to  have  significant  effects  on  outcomes. 

This  explanation  is,  however,  incomplete.  It  is  reasonable  to  argue  that,  given 
sufficient  time  and  resources,  good  policy  and  management  could  be  instituted  to 
address  even  large  numbers  of  factors,  assuming  they  are  amenable  to  policy  and 
management  (i.e.,  well-characterized  and  relatively  stable  across  time  and 
programs).  That  this  has  not  occurred  implies  that  these  non-PM  factors  are 
idiosyncratic  and  unstable,  that  is,  that  every  program  is  unique  in  the  factors  that 
determine  its  outcomes. 


Every  program  is  different.  This  study  has  implicitly  assumed  that  MDAPs 
represent  a  homogeneous  population.  Likewise,  approaches  to  PM 
professionalization  assume  that  programs  are  sufficiently  similar  to  allow  for 
standardized  training  and  for  experiences  in  one  program  to  be  managerially  useful 
in  others.  It  may  be  argued,  however,  that  all  programs  vary  in  their  major  features 
(e.g.,  requirements,  technical  characteristics,  contractors)  as  well  as  in  the  major 
features  of  the  environment  in  which  they  are  acquired  (e.g.,  political  support, 
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economic  circumstances,  industrial  capacity).  Environmental  instabilities  cause 
program  instabilities.  Because  each  program  faces  unique  and  changing 
configurations  of  highly  complex  issues — whether  operational,  economic,  political,  or 
technological — each  program  differs  in  the  extent  to  which  it  is  open  or  amenable  to 
management  throughout  its  life. 

Because  the  circumstances  of  each  program  are  complex,  unique,  and 
unstable,  so  too  are  the  reasons  for  success  and  failure.  Preston  (201 1 )  described 
the  MX  missile  program  as  follows: 

It  was  deemed  as  one  of  the  most  successful  programs  that  the  Air  Force  had 
ever  had.  Why  was  that?  It  was  because  Congress  supported  the  program, 
[DoD]  supported  the  program,  it  got  all  the  funding  it  needed,  and  the 
technology  was  developed  in  time.  ...  You  could  have  the  best  program 
manager  in  the  world  [but]  if  the  contractors  can’t  get  the  technology 
breakthroughs  that  they  need  [or]  if  Congress  changes  its  mind  halfway 
through  a  program  and  cuts  the  funding  ...  you  are  going  to  have  an 
unsuccessful  program. 

If  valid,  this  perspective  helps  explain  the  poor  predictive  power  of  this  study’s 
models. 

Past  policies  inhibit  PM  professionalism.  The  earlier  background  discussion 
touched  on  policy-makers’  desires  to  make  program  management  a  professional 
career  field  that  would  attract  talented  candidates  for  membership.  This  presumably 
would  entail  actions  such  as  establishing  a  professional  body  of  knowledge, 
standards  for  admittance  and  progression,  and  a  code  of  professional  conduct. 

Such  features  would  be  the  “ties  that  bind”  members  of  the  profession  together.  In 
attempting  to  hold  PMs  accountable  for  program  outcomes,  however,  policy-makers 
have  diverted  PMs’  focus  away  from  professional  considerations  and  toward  the 
specific  features  of  their  individual  programs.  To  the  extent  that  each  program  is 
unique,  PMs  have  little  in  common  upon  which  to  build  a  professional  career  field. 

These  conclusions  have  major  implications  for  recommendations  regarding 
PMs,  as  will  be  discussed  in  the  following  section. 
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B.  Recommendations 


With  over  thirty-five  years  of  active  service  in  the  DoD,  the  author  has  no 
idealistic  expectations  regarding  the  likelihood  of  promoting  and  implementing 
significant  changes  where  PMs  are  concerned.  Nevertheless,  with  the  hope  that 
policy-makers  in  both  the  Congress  and  the  DoD  might  seriously  consider  them,  he 
offers  the  following  recommendations  that  follow  from  the  previous  discussion. 

Revise  expectations  of  PM  performance.  If  PMs  indeed  can  have  little  to  no 
influence  over  program  outcomes,  it  is  unreasonable  to  attempt  to  hold  them 
accountable  for  those  outcomes.  Appropriate  perspectives  are  needed  as  to  what  a 
PM  can  be  expected  to  accomplish,  along  with  criteria  for  performance  evaluations 
(see,  for  example,  Fox  &  Miller,  2006,  pp.  177-196).  The  earlier  comments  by 
Preston  (2011)  and  Soloway  (201 1 )  describing  what  a  PM  really  manages  (e.g., 
people,  budgets,  politics,  contracts)  indicate  some  possibilities  for  areas  in  which  the 
PM  could  have  both  responsibility  and  accountability,  for  example,  administration 
and  leadership  of  the  program  management  office;  quality  of  budget  submissions; 
oversight  of  contractual  arrangements;  and  actions  to  enhance  the  program’s 
political  climate. 

Promote  shared  program  ownership  and  accountability.  In  the  past,  policy¬ 
makers  have  treated  the  PM  position  as  an  object  for  improvement:  if  the  PM’s 
professionalism  could  be  improved,  program  outcomes  would  improve.  This  has  the 
obvious  effect  of  deflecting  accountability  from  its  proper  place  and  onto  an  easy 
target — the  PM.  The  notion  of  the  PM  as  the  “single  point  of  contact”  was  exposed 
earlier  as  reflecting  policy-makers’  frustrations  with  having  no  one  to  blame  for  poor 
outcomes  and,  as  indicated  by  the  subsequent  analysis,  it  has  no  correlation  with 
those  outcomes.  Therefore,  if  program  outcomes  are  attributable  to  many 
participants  and  stakeholders,  policy-makers  should  seek  to  promote  a  shared 
sense  of  ownership  for  those  programs,  for  the  sake  of  both  managerial 
effectiveness  and  accountability.  Ideally,  this  would  expand  the  extent  of 
communication  and  collaboration  in  programs,  as  well  as  responsibility  for  them. 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  82  - 

NAVAL  POSTGRADUATE  SCHOOL 


Promote  contingency  approaches  for  PM  career  management.  The  unique 
and  changing  nature  of  each  acquisition  program  argues  against  a  “one-size-fits-all” 
approach  to  managing  PMs.  It  reduces  emphasis  on  minimal  qualification  standards 
and  elevates  emphasis  on  experiential  diversity.  One  senior  career  management 
official  related  the  view  that  PMs  are  best  prepared  for  their  jobs  through  a  range  of 
diverse  educational  and  practical  experiences.  This  official,  who  chose  to  remain 
anonymous,  used  the  metaphor  of  a  spider  web  as  an  illustration  that  many  different 
strands  (career  experiences)  can  lead  to  the  center  of  the  web,  which  represents  a 
professionally  qualified  PM. 

Selection  for  PM  might  be  accomplished  according  to  “best  fit”  rather  than 
“best  qualified,”  where  fit  examines  the  extent  of  alignment  between  the  unique 
characteristics  of  candidate  and  program,  in  addition  to  conformance  with  some 
minimal  qualification  standards.  Selection  should  be  accomplished  at  the  level  at 
which  that  determination  of  fit  is  best  made. 

While  the  analysis  suggested  a  positive  effect  on  outcomes  with  civilian  PMs, 
the  results  were  mixed.  Thus,  there  is  no  strong  basis  for  preferring  one  status  (i.e. , 
military  or  civilian)  over  another  for  PM,  though  it  seems  reasonable  to  ensure 
representation  of  both  in  the  program’s  management  structures.  Whether  that 
balance  results  from  a  military  PM  with  a  civilian  deputy  or  vice  versa  may  likely  be 
determined  more  by  personnel  management  constraints  rather  than  by  any  other 
factors.21 

Because  no  significant  correlation  between  PM  tenure  and  outcomes  was 
found,  the  past  emphasis  on  tenure  is  misplaced.  A  contingency  approach  to  PM 


21  Previous  discussion  indicated  the  difficulty  in  making  changes  to  the  military  personnel  system. 
Regarding  civilians,  one  senior  career  management  official  who  chose  to  remain  anonymous  noted 
that  the  low  numbers  of  civilian  PMs  was  due  to  low  numbers  of  applicants  for  those  positions,  often 
because  civilians  do  not  want  to  relocate  to  the  locality  of  the  PM’s  office.  Uniformed  officers,  in 
contrast,  are  inured  to  frequent  moves. 
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career  management  would  focus,  rather,  on  ensuring  that,  at  any  given  time,  a 
proper  balance  of  fresh  perspectives  and  continuity  is  maintained. 

Retain  decentralized  PM  career  management.  The  history  of  policy  on  PM 
professionalism  reflects  tendencies  toward  centralization  (e.g.,  of  policy  authority) 
and  standardization  (e.g.,  of  qualifications  for  assignment).  In  contrast,  this  paper’s 
analysis  and  arguments  support  decentralized  management  of  the  PM  workforce  so 
as  to  facilitate  contingency  approaches,  such  as  “best  fit”  qualification  and 
assignment  criteria. 

Decentralized  PM  career  management  would  not,  of  course,  be  inconsistent 
with  the  provision  of  centralized  resources  to  enhance  PM  professionalism,  such  as 
those  DoD-wide  training,  education,  and  knowledge-sharing  resources  provided  by 
the  DAU. 

Eliminate  requirements  for  PM  waivers.  Waivers  militate  against  “best  fit” 
assignment  criteria.  Waivers  appear  to  have  no  functions  other  than  as  an 
accountability  measure  and  to  discourage  assignment  of  PMs  who  do  not  meet 
minimum  standards.  Waiver  requirements  implicitly  call  into  question  the  judgment 
of  those  who  request  and  approve  waivers. 

Do  not  abandon  PM  professionalization.  Finally,  this  study  should  not  be 
read  as  an  argument  that  investments  in  PM  professionalization  should  not  continue. 
The  author  has  no  doubt  that  PMs  have  significant  influences — sometimes  good  and 
sometimes  bad — over  their  programs,  only  that  the  traditional  ways  of  viewing  those 
influences  are  inadequate.  The  author  often  states  the  view  that  defense  acquisition 
PMs  have  “the  hardest  job  in  the  world”  and,  accordingly,  they  merit  the  best 
preparation  possible  for  that  job. 
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